[CI] Automatically precompile SPRAL when a new release is tagged

[amontoison]

I believe adding a GitHub Actions script that triggers only on new release tags should be straightforward. It would execute the following commands and upload the resulting archives:

julia --color=yes build_tarballs.jl x86_64-linux-gnu-libgfortran5 --verbose
julia --color=yes build_tarballs.jl x86_64-w64-mingw32-libgfortran5 --verbose
julia --color=yes build_tarballs.jl x86_64-apple-darwin-libgfortran5 --verbose
julia --color=yes build_tarballs.jl aarch64-apple-darwin-libgfortran5 --verbose
julia generate_binaries.jl

If it works, we can reuse the same script for other repositories like GALAHAD, CUTEst or SIFDecode.

The content of build_tarballs.jl is:

# Platforms:
#   - aarch64-apple-darwin
#   - aarch64-linux-gnu
#   - aarch64-linux-musl
#   - armv6l-linux-gnueabihf
#   - armv6l-linux-musleabihf
#   - armv7l-linux-gnueabihf
#   - armv7l-linux-musleabihf
#   - i686-linux-gnu
#   - i686-linux-musl
#   - i686-w64-mingw32
#   - powerpc64le-linux-gnu
#   - x86_64-apple-darwin
#   - x86_64-linux-gnu
#   - x86_64-linux-musl
#   - x86_64-unknown-freebsd
#   - x86_64-w64-mingw32
#
# Targets:
#   - platform-libgfortran3
#   - platform-libgfortran4
#   - platform-libgfortran5
#
# Examples:
# julia --color=yes build_tarballs.jl x86_64-linux-gnu-libgfortran5 --verbose
# julia --color=yes build_tarballs.jl x86_64-w64-mingw32-libgfortran5 --verbose --debug
# julia --color=yes build_tarballs.jl x86_64-apple-darwin-libgfortran5 --verbose
# julia --color=yes build_tarballs.jl aarch64-apple-darwin-libgfortran5 --verbose
using BinaryBuilder, Pkg

name = "SPRAL"
version = v"2023.11.15"

# Collection of sources required to complete build
sources = [
    GitSource("https://github.com/ralna/spral.git", "e723071ce2e0e6181bb65e1b365dc47449e1a912")
]

# Bash recipe for building across all platforms
script = raw"""
# Export dependencies
mkdir ${prefix}/deps
cd ${libdir}
for file in $(ls .); do
   if [[ -f $file ]]; then
      if [[ -z $(ls -la $file | grep 'artifacts') ]]; then
         cp -P ${file} ${prefix}/deps/${file}
      else
         cp -L ${file} ${prefix}/deps/${file}
      fi
   fi
done
cd ${prefix}
cp -rL share/licenses deps/licenses
chmod -R u=rwx deps
tar -czvf deps.tar.gz deps
rm -r deps

# Install a version of Meson ≥ 0.63.0
python3 -m pip install --user --upgrade meson

cd ${WORKSPACE}/srcdir/spral

if [[ "${target}" == *mingw* ]]; then
  HWLOC="hwloc-15"
else
  HWLOC="hwloc"
fi

meson setup builddir --cross-file=${MESON_TARGET_TOOLCHAIN%.*}_gcc.meson \
                     --prefix=$prefix -Dlibhwloc=$HWLOC \
                     -Dlibblas=openblas -Dliblapack=openblas \
                     -Dexamples=true -Dtests=true

for i in {1..10}
do
    meson compile -C builddir || true
done
meson install -C builddir
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = supported_platforms()
platforms = expand_gfortran_versions(platforms)

# The products that we will ensure are always built
products = [
    LibraryProduct("libspral", :libspral)
]

# Dependencies that must be installed before this package can be built
dependencies = [
    Dependency(PackageSpec(name="METIS_jll", uuid="d00139f3-1899-568f-a2f0-47f597d42d70")),
    Dependency(PackageSpec(name="CompilerSupportLibraries_jll", uuid="e66e0078-7015-5450-92f7-15fbd957f2ae")),
    Dependency(PackageSpec(name="OpenBLAS32_jll", uuid="656ef2d0-ae68-5445-9ca0-591084a874a2")),
    Dependency(PackageSpec(name="Hwloc_jll", uuid="e33a78d0-f292-5ffc-b300-72abe9b543c8")),
]

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies; julia_compat="1.6")

The content of generate_binaries.jl is:

# Version
version = "2023.11.15"

platforms = [
   ("aarch64-apple-darwin-libgfortran5"  , "lib", "dylib"),
#  ("aarch64-linux-gnu-libgfortran3"     , "lib", "so"   ),
#  ("aarch64-linux-gnu-libgfortran4"     , "lib", "so"   ),
#  ("aarch64-linux-gnu-libgfortran5"     , "lib", "so"   ),
#  ("aarch64-linux-musl-libgfortran3"    , "lib", "so"   ),
#  ("aarch64-linux-musl-libgfortran4"    , "lib", "so"   ),
#  ("aarch64-linux-musl-libgfortran5"    , "lib", "so"   ),
#  ("powerpc64le-linux-gnu-libgfortran3" , "lib", "so"   ),
#  ("powerpc64le-linux-gnu-libgfortran4" , "lib", "so"   ),
#  ("powerpc64le-linux-gnu-libgfortran5" , "lib", "so"   ),
#  ("x86_64-apple-darwin-libgfortran3"   , "lib", "dylib"),
#  ("x86_64-apple-darwin-libgfortran4"   , "lib", "dylib"),
   ("x86_64-apple-darwin-libgfortran5"   , "lib", "dylib"),
#  ("x86_64-linux-gnu-libgfortran3"      , "lib", "so"   ),
#  ("x86_64-linux-gnu-libgfortran4"      , "lib", "so"   ),
   ("x86_64-linux-gnu-libgfortran5"      , "lib", "so"   ),
#  ("x86_64-linux-musl-libgfortran3"     , "lib", "so"   ),
#  ("x86_64-linux-musl-libgfortran4"     , "lib", "so"   ),
#  ("x86_64-linux-musl-libgfortran5"     , "lib", "so"   ),
#  ("x86_64-unknown-freebsd-libgfortran3", "lib", "so"   ),
#  ("x86_64-unknown-freebsd-libgfortran4", "lib", "so"   ),
#  ("x86_64-unknown-freebsd-libgfortran5", "lib", "so"   ),
#  ("x86_64-w64-mingw32-libgfortran3"    , "bin", "dll"  ),
#  ("x86_64-w64-mingw32-libgfortran4"    , "bin", "dll"  ),
   ("x86_64-w64-mingw32-libgfortran5"    , "bin", "dll"  ),
]

for (platform, libdir, ext) in platforms

  tarball_name = "SPRAL.v$version.$platform.tar.gz"

  if isfile("products/$(tarball_name)")
    # Unzip the tarball generated by BinaryBuilder.jl
    isdir("products/$platform") && rm("products/$platform", recursive=true)
    mkdir("products/$platform")
    run(`tar -xzf products/$(tarball_name) -C products/$platform`)

    if isfile("products/$platform/deps.tar.gz")
      # Unzip the tarball of the dependencies
      run(`tar -xzf products/$platform/deps.tar.gz -C products/$platform`)

      # Copy the license of each dependency
      for folder in readdir("products/$platform/deps/licenses")
        cp("products/$platform/deps/licenses/$folder", "products/$platform/share/licenses/$folder")
      end
      rm("products/$platform/deps/licenses", recursive=true)

      # Copy the shared library of each dependency
      for file in readdir("products/$platform/deps")
        any(endswith.(file, [ext, ".a"])) && cp("products/$platform/deps/$file", "products/$platform/$libdir/$file")
      end

      # Remove the folder used to unzip the tarball of the dependencies
      rm("products/$platform/deps", recursive=true)
      rm("products/$platform/deps.tar.gz", recursive=true)

      # Create the archives for SPRAL_binaries
      isfile("SPRAL_binaries-$version.$platform.tar.gz") && rm("SPRAL_binaries-$version.$platform.tar.gz")
      isfile("SPRAL_binaries-$version.$platform.zip") && rm("SPRAL_binaries-$version.$platform.zip")
      cd("products/$platform")

      # Create a folder with the version number of SPRAL
      mkdir("SPRAL_binaries-$version")
      for folder in ("include", "share", "modules", "lib", "bin", "examples", "tests")
        cp(folder, "SPRAL_binaries-$version/$folder")
      end

      cd("SPRAL_binaries-$version")
      if ext == "dll"
        run(`zip -r --symlinks ../../../SPRAL_binaries-$version.$platform.zip include share modules lib bin examples tests`)
      else
        run(`tar -czf ../../../SPRAL_binaries-$version.$platform.tar.gz include share modules lib bin examples tests`)
      end
      cd("../../..")

      # Remove the folder used to unzip the tarball generated by BinaryBuilder.jl
      rm("products/$platform", recursive=true)
    else
      @warn("The tarball deps.tar.gz is missing in $(tarball_name)!")
    end
  else
    @warn("The tarball for the platform $platform was not generated!")
  end
end

[jfowkes]

Yeah sounds good but we need to fix mesonbuild/meson/issues/12523 first.

[amontoison]

@jfowkes I tested it a little bit this week-end and I have the following result: https://github.com/amontoison/spral/releases

When I tag a new version vx.y.z on my fork amontoison/spral, this action here is triggered. The YAML file for the action is here. It cross-compiles SPRAL on a few platforms and at the end of the process creates a release with the archives.

[jfowkes]

@amontoison is it possible to have an empty body_path? I think the release description will always have to be manually edited. The current ChangeLog is ancient.

[amontoison]

@jfowkes Yes, I checked the README of actions/create-release and you just need to to use body with an empty message instead of body_path. But we have other actions that can generate the release message automatically for us if we want. I just wanted to prototype it yesterday.

I also want to use to GitHub Actions to directly provide the content of the following variables in the Julia files: https://github.com/amontoison/spral/blob/master/.github/julia/build_tarballs.jl#L32 https://github.com/amontoison/spral/blob/master/.github/julia/build_tarballs.jl#L36 https://github.com/amontoison/spral/blob/master/.github/julia/generate_binaries.jl#L2

It should not be to hard. It will help a lot the users if we can use it in SPRAL, GALAHAD, CUTEst and SIFDecode repositories to provide precompiled binaries.

About the ChangeLog, maybe we should remove it?

[jfowkes]

@amontoison sounds good to me. Yes agreed we should probably remove the ChangeLog and rely on the Github releases description instead.

[amontoison]

The culprit was Ninja... I fixed the issue with the two additional lines in the build_tarballs.jl.

python3 -m pip install --user --upgrade ninja
cp /root/.local/bin/ninja /usr/bin/ninja

[jfowkes]

@amontoison could you create a PR for this? I think this is too fancy for me to do...