Use Existing Vendor Tarball Rather Than Creating One on Buildtime

[Pi-Cla]

On openSUSE we create a vendored tarball of rust dependencies before building because the virtual machine which builds the package is intentionally not connected to the internet.

So eventually the configure script attempts to connect to the internet but fails:

[   53s] /usr/bin/mkdir -p completions
[   53s] make[2]: Leaving directory '/home/abuild/rpmbuild/BUILD/sile-0.15.3'
[   64s] make[2]: Entering directory '/home/abuild/rpmbuild/BUILD/sile-0.15.3'
[   64s] set -e
[   64s] export AUTOTOOLS_DEPENDENCIES="src/bin/sile.rs src/lib.rs src/cli.rs Cargo.toml build-aux/build.rs Cargo.lock .version"
[   64s] CARGO_TARGET_DIR=./target /usr/bin/cargo build --verbose --features lua54 --features vendored  --features variations --release --locked
[   64s] CARGO_TARGET_DIR=./target /usr/bin/cargo build --quiet --message-format=json --features lua54 --features vendored  --features variations --release --locked | \
[   64s]        /usr/bin/jq -sr 'map(select(.reason == "build-script-executed")) | last | .out_dir' > ./target/release/.cargo_out_dir
[   64s]     Updating crates.io index
[   64s] warning: spurious network error (3 tries remaining): [6] Couldn't resolve host name (Could not resolve host: index.crates.io)
[   64s] warning: spurious network error (2 tries remaining): [6] Couldn't resolve host name (Could not resolve host: index.crates.io)
[   64s] warning: spurious network error (1 tries remaining): [6] Couldn't resolve host name (Could not resolve host: index.crates.io)
[   64s] error: failed to get `anyhow` as a dependency of package `sile v0.15.3 (/home/abuild/rpmbuild/BUILD/sile-0.15.3)`

Is there a way to both prevent it from connecting to the internet and to point it to the vendored tarball of rust dependencies?

[alerque]

If you run cargo fetch with your target triple ahead of time you should get all the Rust deps you need. I don't know how you go about taring those up, but the Cargo cache location and all is quite standard, we're not doing anything special with that. Thereafter the build can be run offline, for example export CARGO_FEATURE_FLAGS==--offline will make sure and keep Cargo's head out of the clouds during the SILE build process.

[Pi-Cla]

I have tried export CARGO_FEATURE_FLAGS==--offline but it still results in the makefile calling a cargo build command without the --offline flag so it still tries to connect to the internet

[alerque]

Try configuring with --with-system-lua-sources.

I wrote a lot more but the power went out :-(

[Pi-Cla]

Still the same result.

I wonder if there is a way to get the Makefile to only build the lua portion of SILE. Then I could just use the typical openSUSE cargo macros to build with the vendored deps...

[uncomfyhalomacro]

I have fixed it. @Pi-Cla I am pinging you everywhere :P

[uncomfyhalomacro]

I have tried export CARGO_FEATURE_FLAGS==--offline but it still results in the makefile calling a cargo build command without the --offline flag so it still tries to connect to the internet

@Pi-Cla That flag me thinks are for the feature flags for a Rust crate that's probably starts with the name lua. The correct flag to use vendor rust sources in our builds are

unset LIBSSH2_SYS_USE_PKG_CONFIG
export RUSTFLAGS=" -Clink-arg=-Wl,-z,relro,-z,now -C debuginfo=2 -C incremental=false -C strip=none"
export CARGO_AUDITABLE=auditable
export CARGO_FEATURE_VENDORED=1

@alerque btw we have resolved it. you don't need a fix on your side. :saluting_face:

[alerque]

Thanks for all the work on packaging this guys. I'm glad you got it worked out to build (and as a distro packager myself it warms my heart to see the dynamic build with system dependencies, offline build phase, etc.!).

Even though you got it to build, looking at the spec file I do have to wonder if there is anything we can do here upstream to make downstream builds easier to adapt to various requirements. I left a comment over there, but I'm not really clear on how you're using system Lua 5.1 headers but then also setting the vendored feature to get mlua to use LuaJIT anyway? If so why not just use --without-system-lua-sources? Is there a way we could simplify or group the config options in a more idiomatic way that would have been easier to figure out? It's been a bit of a brain teaser to figure out how to support both dynamic linking with system deps and static builds with also optionally everything embedded.

Also on the RUSTFLAGS topic, is there a better way to set the flag we need without forcing you to have to set or reset everything like that?

[uncomfyhalomacro]

Even though you got it to build, looking at the spec file I do have to wonder if there is anything we can do here upstream to make downstream builds easier to adapt to various requirements. I left a comment over there, but I'm not really clear on how you're using system Lua 5.1 headers but then also setting the vendored feature to get mlua to use LuaJIT anyway? If so why not just use --without-system-lua-sources?

TBH I am not really sure either. Vendored dependencies are really new to openSUSE e.g. Rust and has been since 2 years ago (except NPM packages?). I am one of the early adopters of this style of packaging. And we gradually started to support vendoring dependencies to other languages e.g. Zig has started using their own package manager of which I have to accommodate the changes as well.

Also on the RUSTFLAGS topic, is there a better way to set the flag we need without forcing you to have to set or reset everything like that?

Tbh I am not sure either. The RUSTFLAGS I use are from one of the rpm macros which most Rust packages in openSUSE use during cargo build. See https://github.com/Firstyear/cargo-packaging

you're using system Lua 5.1 headers but then also setting the vendored feature to get mlua to use LuaJIT anyway?

@Pi-Cla might clean that up for later (?). I was experimenting a bit and I am not sure which Lua to use.

Even though you got it to build, looking at the spec file I do have to wonder if there is anything we can do here upstream to make downstream builds easier to adapt to various requirements.

Tbh, I really don't know the answer to that. We have to adapt when upstream does changes things and although, sometimes it can be tiring, but so far, I was able to figure out how most software I package for openSUSE are built like this one. And you have already done so much to tell us the sources and what to do in https://github.com/sile-typesetter/sile?tab=readme-ov-file#from-source. What I usually do is check deps and then check your build scripts as well in .github (GitHub Actions).

[Pi-Cla]

@alerque We finally got it to build with the luajit-devel headers. For some reason --I think because the headers are placed in a non-standard location-- we needed to add:

export LUA_INCLUDE="-I/usr/include/luajit-5_1-2.1"

to the %build and %install sections of the specfile.

Unless this is the standard location this looks like openSUSE packaging just placing headers in non-standard locations again... :(

Edit: this also works

export LUA_INCLUDE="$(pkg-config --cflags-only-I luajit)"