search

rust-av / dav1d-rs

libdav1d rust bindings
MIT License
37 stars 16 forks source link

Doesn't support build or static linking #80

Open kornelski opened 12 months ago

kornelski commented 12 months ago

I'm trying to run rav1e's arm64 dav1d tests using static build of libdav1d, but either the system-deps crate isn't working properly, or there's something wrong with its integration.

I'm setting SYSTEM_DEPS_DAV1D_LINK=static and SYSTEM_DEPS_DAV1D_BUILD_INTERNAL=auto (also tried always, same result), and system-deps builds it, but then unnecessarily falls back to pkg-config, misconfigures it, and ends up incorrectly linking dynamically.

https://github.com/kornelski/rav1e/blob/9f257e6870c36efb683ec19d1974db3746113296/.github/workflows/rav1e.yml#L197-L198

2023-10-07T17:40:04.1810201Z ##[group]Run cargo test -vv --lib --verbose --release --target=aarch64-unknown-linux-gnu \
2023-10-07T17:40:04.1810705Z cargo test -vv --lib --verbose --release --target=aarch64-unknown-linux-gnu \
2023-10-07T17:40:04.1811311Z            --no-default-features --features=asm,decode_test_dav1d -- \
2023-10-07T17:40:04.1811633Z            --include-ignored
2023-10-07T17:40:04.1866898Z shell: /usr/bin/bash -e {0}
2023-10-07T17:40:04.1867137Z env:
2023-10-07T17:40:04.1867359Z   RUST_BACKTRACE: full
2023-10-07T17:40:04.1867612Z   RUSTC_WRAPPER: sccache
2023-10-07T17:40:04.1867845Z   SCCACHE_CACHE_SIZE: 300M
2023-10-07T17:40:04.1868117Z   SCCACHE_DIR: /home/runner/.cache/sccache
2023-10-07T17:40:04.1868390Z   SCCACHE_IDLE_TIMEOUT: 0
2023-10-07T17:40:04.1868622Z   CARGO_INCREMENTAL: 0
2023-10-07T17:40:04.1868866Z   CARGO_TERM_COLOR: always
2023-10-07T17:40:04.1869168Z   CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUNNER: qemu-aarch64
2023-10-07T17:40:04.1869547Z   CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS: -Clinker=aarch64-linux-gnu-gcc
2023-10-07T17:40:04.1869996Z   PKG_CONFIG_PATH: /usr/lib/aarch64-linux-gnu/pkgconfig
2023-10-07T17:40:04.1870295Z   PKG_CONFIG_SYSROOT_DIR: /
2023-10-07T17:40:04.1870568Z   SYSTEM_DEPS_DAV1D_BUILD_INTERNAL: auto
2023-10-07T17:40:04.1870843Z   SYSTEM_DEPS_DAV1D_LINK: static
2023-10-07T17:40:04.1871071Z ##[endgroup]
…
2023-10-07T17:42:22.4874510Z [dav1d-sys 0.7.1] [68/148] Generating 'src/libdav1d.a.p/loopfilter16_sse.obj'.
2023-10-07T17:42:24.1448553Z [dav1d-sys 0.7.1] [69/148] Generating 'src/libdav1d.a.p/itx16_avx2.obj'.
2023-10-07T17:42:25.0537801Z [dav1d-sys 0.7.1] [70/148] Generating 'src/libdav1d.a.p/itx16_sse.obj'.
2023-10-07T17:42:25.1853053Z [dav1d-sys 0.7.1] [71/148] Generating 'src/libdav1d.a.p/looprestoration16_sse.obj'.
2023-10-07T17:42:25.3430088Z [dav1d-sys 0.7.1] [72/148] Compiling C object src/libdav1d.a.p/cpu.c.o
…
2023-10-07T17:42:37.1729541Z [dav1d-sys 0.7.1] [144/148] Compiling C object tests/libfuzzer/dav1d_fuzzer_mt.p/dav1d_fuzzer.c.o
2023-10-07T17:42:37.1767906Z [dav1d-sys 0.7.1] [145/148] Linking target tests/libfuzzer/dav1d_fuzzer
2023-10-07T17:42:37.2335633Z [dav1d-sys 0.7.1] [146/148] Linking target tests/libfuzzer/dav1d_fuzzer_mt
2023-10-07T17:42:37.3429104Z [dav1d-sys 0.7.1] [147/148] Compiling C object tests/seek_stress.p/seek_stress.c.o
2023-10-07T17:42:37.4069424Z [dav1d-sys 0.7.1] [148/148] Linking target tests/seek_stress
2023-10-07T17:42:37.6326896Z [dav1d-sys 0.7.1] ninja: Entering directory `/home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/build'
2023-10-07T17:42:37.6921998Z [dav1d-sys 0.7.1] [1/10] Generating include/vcs_version.h with a custom command
2023-10-07T17:42:37.7017954Z [dav1d-sys 0.7.1] Installing src/libdav1d.a to /home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/release/lib/x86_64-linux-gnu
…
2023-10-07T17:42:37.7029154Z [dav1d-sys 0.7.1] Installing /home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/build/meson-private/dav1d.pc to /home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/release/lib/x86_64-linux-gnu/pkgconfig
2023-10-07T17:42:37.7373781Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=DAV1D_NO_PKG_CONFIG
2023-10-07T17:42:37.7374751Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7375619Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7376393Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_ALLOW_CROSS
2023-10-07T17:42:37.7377183Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_ALLOW_CROSS
2023-10-07T17:42:37.7378400Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7379352Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7379910Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
2023-10-07T17:42:37.7380386Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG
2023-10-07T17:42:37.7380988Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7381648Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7382221Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7383162Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7383912Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7384510Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7385029Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
2023-10-07T17:42:37.7385515Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG
2023-10-07T17:42:37.7386085Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7386685Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7387243Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_PATH
2023-10-07T17:42:37.7387748Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_PATH
2023-10-07T17:42:37.7388325Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7388946Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7389521Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_LIBDIR
2023-10-07T17:42:37.7390037Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
2023-10-07T17:42:37.7390615Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7391262Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7391845Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7392384Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7400316Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7401216Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7402024Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7402778Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG
2023-10-07T17:42:37.7403467Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG
2023-10-07T17:42:37.7404113Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7404798Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_PATH_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7405542Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_PATH
2023-10-07T17:42:37.7406906Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_PATH
2023-10-07T17:42:37.7407563Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7408267Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7409010Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_LIBDIR
2023-10-07T17:42:37.7409893Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
2023-10-07T17:42:37.7410490Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64-unknown-linux-gnu
2023-10-07T17:42:37.7411065Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_aarch64_unknown_linux_gnu
2023-10-07T17:42:37.7411579Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=TARGET_PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7412038Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
2023-10-07T17:42:37.7424476Z [dav1d-sys 0.7.1] cargo:rustc-link-search=native=//home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/release/lib/x86_64-linux-gnu
2023-10-07T17:42:37.7425316Z [dav1d-sys 0.7.1] cargo:rustc-link-lib=dav1d
2023-10-07T17:42:37.7425811Z [dav1d-sys 0.7.1] cargo:rustc-link-lib=dl
2023-10-07T17:42:37.7426588Z [dav1d-sys 0.7.1] cargo:include=//home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/release/include
2023-10-07T17:42:37.7427557Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_BUILD_INTERNAL
2023-10-07T17:42:37.7428396Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_LINK
2023-10-07T17:42:37.7431125Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_LIB
2023-10-07T17:42:37.7436576Z      Running `CARGO=/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/cargo CARGO_CRATE_NAME=dav1d_sys CARGO_MANIFEST_DIR=/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dav1d-sys-0.7.1 CARGO_PKG_AUTHORS='Luca Barbato <lu_zero@gentoo.org>' CARGO_PKG_DESCRIPTION='FFI bindings to dav1d' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE=MIT CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=dav1d-sys CARGO_PKG_README='' CARGO_PKG_REPOSITORY='https://github.com/rust-av/dav1d-rs' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.7.1 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=7 CARGO_PKG_VERSION_PATCH=1 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/runner/work/rav1e/rav1e/target/release/deps:/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib:/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib' OUT_DIR=/home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out sccache /home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name dav1d_sys --edition=2021 /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/dav1d-sys-0.7.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C debuginfo=2 -C metadata=30816a30f6c0dc33 -C extra-filename=-30816a30f6c0dc33 --out-dir /home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/deps --target aarch64-unknown-linux-gnu -L dependency=/home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/deps -L dependency=/home/runner/work/rav1e/rav1e/target/release/deps --extern libc=/home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/deps/liblibc-7740b950ce1d20d0.rmeta --cap-lints warn -Clinker=aarch64-linux-gnu-gcc -L native=//home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-2f9630ea28931b6d/out/dav1d/release/lib/x86_64-linux-gnu -l dav1d -l dl --cfg system_deps_have_dav1d`
2023-10-07T17:42:37.7440218Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_LIB_FRAMEWORK
2023-10-07T17:42:37.7440932Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_SEARCH_NATIVE
2023-10-07T17:42:37.7443604Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_SEARCH_FRAMEWORK
2023-10-07T17:42:37.7444530Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_INCLUDE
2023-10-07T17:42:37.7445226Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_NO_PKG_CONFIG
2023-10-07T17:42:37.7445944Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_BUILD_INTERNAL
2023-10-07T17:42:37.7446553Z [dav1d-sys 0.7.1] cargo:rerun-if-env-changed=SYSTEM_DEPS_DAV1D_LINK
2023-10-07T17:42:37.7447038Z [dav1d-sys 0.7.1] 
2023-10-07T17:42:37.7447636Z [dav1d-sys 0.7.1] cargo:rustc-cfg=system_deps_have_dav1d
kornelski commented 12 months ago

Full log 19_Run dav1d tests (arm64).txt.zip

lu-zero commented 12 months ago

This is quite annoying, @gdesmott do you have any idea?

gdesmott commented 11 months ago

@gdesmott do you have any idea?

Not really. One should debug what's system-dep is doing wrong here (check the env variables defined) and then open a ticket explaining what's going wrong.

lu-zero commented 11 months ago

I hope it is a local misuse, but I wanted to double-check in case it is a known issue.

gdesmott commented 11 months ago

Maybe it's related to https://github.com/gdesmott/system-deps/pull/77 ?

lu-zero commented 5 months ago

I'm bumping dav1d-sys to use system-deps 6.2.1, @kornelski do you have time to check if it solves you problem? (also if you want to make a PR to add the cross compilation target to our CI I'd land it)

kornelski commented 5 months ago

That build is still failing:

https://github.com/kornelski/rav1e/actions/runs/8630802947/job/23657875900

 note: /usr/lib/gcc-cross/aarch64-linux-gnu/12/../../../../aarch64-linux-gnu/bin/ld: skipping incompatible //home/runner/work/rav1e/rav1e/target/aarch64-unknown-linux-gnu/release/build/dav1d-sys-28eeb49e6126283b/out/dav1d/release/lib/x86_64-linux-gnu/libdav1d.a when searching for -ldav1d
          /usr/lib/gcc-cross/aarch64-linux-gnu/12/../../../../aarch64-linux-gnu/bin/ld: cannot find -ldav1d: No such file or directory
          collect2: error: ld returned 1 exit status
lu-zero commented 5 months ago

Sigh, I'll cut the release for the API update today and we'll see what's wrong with cross compilation.