`include_bytes` does not work as expected with`buildRustPackage` for git dependencies

l-monninger commented 1 month ago

Describe the bug

When using build buildRustPackage to build a crate that has a git dependency which uses include_bytes, the dependency fails to compile.

Ostensibly, this is a consequence of some setup in buildRustPackage.

Steps To Reproduce

Steps to reproduce the behavior:

Clone this project and checkout nix-issue/rustBuildPackage (sorry for the naming)
nix --extra-experimental-features "nix-command flakes" --option filter-syscalls false build '.#m1-da-light-node

Note: I have overwrittenbuildPhase to counteract file writability issues which I was initially under the impression buildRustPackage would handle.

buildPhase = ''
        export HOME=$(mktemp -d)
        export RUSTFLAGS="${RUSTFLAGS}"
        cargo clean
        cargo build --release
    '';

Expected behavior

I would expect deps using include_bytes to compile without error if they would in other environments.

Notify maintainers

@dasJ

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
  /nix/store/slh7g2ahqxc969cizlbva4w4yad53fyy-DarwinTools-1
  /nix/store/ahqm7bbi24kzi9znjvkbaa2c3c564nfz-adv_cmds-119-locale
  /nix/store/5wgl1kh8klm45pv4g55k5fw9ffdyz070-apple-framework-CoreFoundation-11.0.0
  /nix/store/5g8x9hsry91a19lvnsq7gvcxira91nf1-bash-interactive-5.2p26
  /nix/store/mkhbd9zmd5szdgcbziw787954jpq0c72-binutils-2.41
  /nix/store/z6cdli0clfzab9qf43h2mi604sv72zb8-binutils-2.41-lib
  /nix/store/c2k1zn42616gmd0ychchkwrbxybgwdrg-bzip2-1.0.8-bin
  /nix/store/jg7372g913h1r8d2jvi7fpbjim8ws0wq-cctools-binutils-darwin-16.0.6-973.0.1
  /nix/store/qz21f7s43wbs4g3xz4f2lhwqnkahja0v-cctools-binutils-darwin-wrapper-16.0.6-973.0.1
  /nix/store/v1v5hh3k07i91yflxyycj385rh64v1c8-cctools-llvm-16.0.6-973.0.1
  /nix/store/4lwsbgvbrvx1m83w73ygyzxih2q6pijh-cctools-port-973.0.1
  /nix/store/ww7lviar1xyd5az6y763yp5wigvwn3m8-clang-16.0.6
  /nix/store/4s2s478cgc19jclm1zg2anjd0kh549nb-clang-16.0.6-lib
  /nix/store/s7lwib0lr80bqq0k43vd1ihr63yc6yql-clang-wrapper-16.0.6
  /nix/store/k0f2w8rm0bz5c7bjmfv76j2f11garsgd-compiler-rt-libc-16.0.6
  /nix/store/nlz9am9p24zxlngxhli5fy41888jgvz9-compiler-rt-libc-16.0.6-dev
  /nix/store/x07qawlqzqac7mz7cq0k8q9fks6bkpgc-coreutils-9.4
  /nix/store/j9wgjc9cydqihgkma0ss3g068r9q16x8-diffutils-3.10
  /nix/store/iyv4s7bkzzmksa3nvdpkdw4wlxa4isa3-ed-1.20.1
  /nix/store/v9c80jbiljy4ks8bp03wrwmskgpjv8j5-expand-response-params
  /nix/store/yzac7wx4aa5wn936w9inbjq7804ckrcb-file-5.45
  /nix/store/a6svx1a2mxil699812p3rpzi18p9zjwg-findutils-4.9.0
  /nix/store/w5sfbsjxf3hhkwajqyq62n2mng58ap99-gawk-5.2.2
  /nix/store/6m4k6vg2w6p8mb422g8s0mi3nkirwp09-gnu-config-2023-09-19
  /nix/store/3xmkcbz15975z0q1nfk45whf95x3fw4w-gnugrep-3.11
  /nix/store/b8fn7032mmzhyzw4ykbjwd0qfwxwr2ds-gnumake-4.4.1
  /nix/store/pbgsjg2pqpqlik3pvfdrnwn5k362bwi2-gnused-4.9
  /nix/store/79jzfdixb8jvl7791y90f8r2ywbk4snq-gnutar-1.35
  /nix/store/zwkrys0d42rz7mapym3q43q3j5xx7q00-gzip-1.13
  /nix/store/16kbg4bnbknh30k9jql69gx8b9a910cz-libSystem-11.0.0
  /nix/store/iv35kw90sl8m882hkc2nwgk6czn30hcf-libcxx-16.0.6-dev
  /nix/store/s7mqbh4zgjq9y89a79134rpfv1db5zac-libtapi-1100.0.11
  /nix/store/3slacif3jgwq7wgk64r4pxjhh6f0knd3-lld-16.0.6
  /nix/store/2mpxplfk2ymnf0zlgdf0ly5qs9qbpqxh-llvm-16.0.6
  /nix/store/8wc3q4jq5506n88mfcfgvqp1xv7japff-llvm-16.0.6-lib
  /nix/store/zq8j5a8pskszvpr6hsnbg1in9n2ppcww-llvm-binutils-16.0.6
  /nix/store/qxqqwhfjyagwdrijlwrpqg4jx9hrlr71-ncurses-6.4
  /nix/store/kxniwccp3ymygfbzx33gwbyp9823saw0-nix-info
  /nix/store/v7jfrx1adr5c8mr45iw2y623axjac00g-patch-2.7.6
  /nix/store/k80nj1xanamk6qilc20zafyz86ggxc1a-pcre2-10.43
  /nix/store/vim8nh0xiybdapifjz6jsyzl8c1cgy60-post-link-sign-hook
  /nix/store/x32b5cyz679r42hsv2axjbv7ni70hcvg-readline-8.2p10
  /nix/store/k1shz0bgk4frqg92x4inzb75i6xx3la1-signing-utils
  /nix/store/6wj08vpv6qyv1q5zpqakx0kh44bnisks-sigtool-0.1.3
  /nix/store/9kswmdwjwz05aawgcbxg9krsxq00rgih-stdenv-darwin
  /nix/store/j2v3p8x54394qvrhxg2dbpvcckvqnay0-update-autotools-gnu-config-scripts-hook
  /nix/store/4ii5r8v349adjnmw1ldvp1xd2c33ff87-xz-5.4.6-bin
copying path '/nix/store/slh7g2ahqxc969cizlbva4w4yad53fyy-DarwinTools-1' from 'https://cache.nixos.org'...
copying path '/nix/store/ahqm7bbi24kzi9znjvkbaa2c3c564nfz-adv_cmds-119-locale' from 'https://cache.nixos.org'...
copying path '/nix/store/b8fn7032mmzhyzw4ykbjwd0qfwxwr2ds-gnumake-4.4.1' from 'https://cache.nixos.org'...
copying path '/nix/store/pbgsjg2pqpqlik3pvfdrnwn5k362bwi2-gnused-4.9' from 'https://cache.nixos.org'...
copying path '/nix/store/qxqqwhfjyagwdrijlwrpqg4jx9hrlr71-ncurses-6.4' from 'https://cache.nixos.org'...
copying path '/nix/store/v9c80jbiljy4ks8bp03wrwmskgpjv8j5-expand-response-params' from 'https://cache.nixos.org'...
copying path '/nix/store/16kbg4bnbknh30k9jql69gx8b9a910cz-libSystem-11.0.0' from 'https://cache.nixos.org'...
copying path '/nix/store/k80nj1xanamk6qilc20zafyz86ggxc1a-pcre2-10.43' from 'https://cache.nixos.org'...
copying path '/nix/store/5wgl1kh8klm45pv4g55k5fw9ffdyz070-apple-framework-CoreFoundation-11.0.0' from 'https://cache.nixos.org'...
copying path '/nix/store/c2k1zn42616gmd0ychchkwrbxybgwdrg-bzip2-1.0.8-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/k0f2w8rm0bz5c7bjmfv76j2f11garsgd-compiler-rt-libc-16.0.6' from 'https://cache.nixos.org'...
copying path '/nix/store/x07qawlqzqac7mz7cq0k8q9fks6bkpgc-coreutils-9.4' from 'https://cache.nixos.org'...
copying path '/nix/store/yzac7wx4aa5wn936w9inbjq7804ckrcb-file-5.45' from 'https://cache.nixos.org'...
copying path '/nix/store/79jzfdixb8jvl7791y90f8r2ywbk4snq-gnutar-1.35' from 'https://cache.nixos.org'...
copying path '/nix/store/w5sfbsjxf3hhkwajqyq62n2mng58ap99-gawk-5.2.2' from 'https://cache.nixos.org'...
copying path '/nix/store/zwkrys0d42rz7mapym3q43q3j5xx7q00-gzip-1.13' from 'https://cache.nixos.org'...
copying path '/nix/store/z6cdli0clfzab9qf43h2mi604sv72zb8-binutils-2.41-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/iyv4s7bkzzmksa3nvdpkdw4wlxa4isa3-ed-1.20.1' from 'https://cache.nixos.org'...
copying path '/nix/store/6m4k6vg2w6p8mb422g8s0mi3nkirwp09-gnu-config-2023-09-19' from 'https://cache.nixos.org'...
copying path '/nix/store/iv35kw90sl8m882hkc2nwgk6czn30hcf-libcxx-16.0.6-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/6wj08vpv6qyv1q5zpqakx0kh44bnisks-sigtool-0.1.3' from 'https://cache.nixos.org'...
copying path '/nix/store/4ii5r8v349adjnmw1ldvp1xd2c33ff87-xz-5.4.6-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/v7jfrx1adr5c8mr45iw2y623axjac00g-patch-2.7.6' from 'https://cache.nixos.org'...
copying path '/nix/store/j2v3p8x54394qvrhxg2dbpvcckvqnay0-update-autotools-gnu-config-scripts-hook' from 'https://cache.nixos.org'...
copying path '/nix/store/3xmkcbz15975z0q1nfk45whf95x3fw4w-gnugrep-3.11' from 'https://cache.nixos.org'...
copying path '/nix/store/mkhbd9zmd5szdgcbziw787954jpq0c72-binutils-2.41' from 'https://cache.nixos.org'...
copying path '/nix/store/a6svx1a2mxil699812p3rpzi18p9zjwg-findutils-4.9.0' from 'https://cache.nixos.org'...
copying path '/nix/store/j9wgjc9cydqihgkma0ss3g068r9q16x8-diffutils-3.10' from 'https://cache.nixos.org'...
copying path '/nix/store/nlz9am9p24zxlngxhli5fy41888jgvz9-compiler-rt-libc-16.0.6-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/kxniwccp3ymygfbzx33gwbyp9823saw0-nix-info' from 'https://cache.nixos.org'...
copying path '/nix/store/s7mqbh4zgjq9y89a79134rpfv1db5zac-libtapi-1100.0.11' from 'https://cache.nixos.org'...
copying path '/nix/store/8wc3q4jq5506n88mfcfgvqp1xv7japff-llvm-16.0.6-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/x32b5cyz679r42hsv2axjbv7ni70hcvg-readline-8.2p10' from 'https://cache.nixos.org'...
copying path '/nix/store/5g8x9hsry91a19lvnsq7gvcxira91nf1-bash-interactive-5.2p26' from 'https://cache.nixos.org'...
copying path '/nix/store/4lwsbgvbrvx1m83w73ygyzxih2q6pijh-cctools-port-973.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/4s2s478cgc19jclm1zg2anjd0kh549nb-clang-16.0.6-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/3slacif3jgwq7wgk64r4pxjhh6f0knd3-lld-16.0.6' from 'https://cache.nixos.org'...
copying path '/nix/store/2mpxplfk2ymnf0zlgdf0ly5qs9qbpqxh-llvm-16.0.6' from 'https://cache.nixos.org'...
copying path '/nix/store/zq8j5a8pskszvpr6hsnbg1in9n2ppcww-llvm-binutils-16.0.6' from 'https://cache.nixos.org'...
copying path '/nix/store/ww7lviar1xyd5az6y763yp5wigvwn3m8-clang-16.0.6' from 'https://cache.nixos.org'...
copying path '/nix/store/v1v5hh3k07i91yflxyycj385rh64v1c8-cctools-llvm-16.0.6-973.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/jg7372g913h1r8d2jvi7fpbjim8ws0wq-cctools-binutils-darwin-16.0.6-973.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/vim8nh0xiybdapifjz6jsyzl8c1cgy60-post-link-sign-hook' from 'https://cache.nixos.org'...
copying path '/nix/store/k1shz0bgk4frqg92x4inzb75i6xx3la1-signing-utils' from 'https://cache.nixos.org'...
copying path '/nix/store/qz21f7s43wbs4g3xz4f2lhwqnkahja0v-cctools-binutils-darwin-wrapper-16.0.6-973.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/s7lwib0lr80bqq0k43vd1ihr63yc6yql-clang-wrapper-16.0.6' from 'https://cache.nixos.org'...
copying path '/nix/store/9kswmdwjwz05aawgcbxg9krsxq00rgih-stdenv-darwin' from 'https://cache.nixos.org'...
 - system: `"aarch64-darwin"`
 - host os: `Darwin 23.2.0, macOS 14.2.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.20.3`
 - nixpkgs: `/Users/l-monninger/.nix-defexpr/channels/nixpkgs`

Add a :+1: reaction to issues you find important.

johnbchron commented 1 month ago

I'm getting a FOD hash mismatch error. Are you sure things are up to date? Maybe the Cargo.lock is out of date?

error: hash mismatch in fixed-output derivation '/nix/store/a76g3p1ph5aljw2v8qkyiz28b5qm3xm7-aptos-core-e9b4212.drv':
         specified: sha256-53ObE7yoEMuZWjIAXXAm4hDBBKU1VhgEj/Zc9EQ4MBA=
            got:    sha256-uT/HGPsUhYDrOp4/fhRiRh5Jmi48zHLw7n9tR6Iupdo=

EDIT: I ran this without cloning using the following, so let me know if I should clone it instead (though it shouldn't matter)

nix build 'github:movementlabsxyz/movement?branch=nix-issue/rustBuildPackage#m1-da-light-node' -L

radupopa369 commented 1 month ago

The thing is that I can build the project with cargo build . So Cargo.lock must be fine

PJungkamp commented 1 month ago

Why is there both a cargoSha256 and a cargoLock in your derivation? Since cargoSha256 is incompatible with git dependencies you should remove it. Your ed22519-dalek patch to the git version is in the [[patch.unused]] section in your Cargo.lock your lockfile still uses the non-git version of ed25519 in it's actual [[package]] section. Did you really run cargo update on that repo? It should warn you if your patch isn't used at all.

The rest of the derivation is also weird.

Why are you changing the buildPhase? Especially cargo clean should not be necessary and all cargo commands should include a --frozen. I think you should remove your buildPhase alltogether.
Why is pkg-config twice in your buildInputs? It's only invoked at compile time for package discovery and thus belongs into nativeBuildInputs and should not be in buildInputs at all.

I think the derivation is seriously messed up.

EDIT:

I was to curious on that error and it's just terrible code....

The ed25519-dalek crates.io patch is unused.
The outputHashes for multiple dependencies were out of date.
nativeBuildInputs wasn't used where appropriate.
The custom buildPhase was seriously broken.

After I rewrote/fixed the nix/m1-da-light-node.nix in that project I've gotten to this error:

   Compiling move-prover v0.1.0 (https://github.com/movementlabsxyz/aptos-core?rev=e9b42128f8ed51e90d06beec72d32797693ab66c#e9b42128)
error: couldn't read /build/cargo-vendor-dir/move-prover-0.1.0/src/../../../../aptos-move/framework/src/aptos-natives.bpl: No such file or directory (os error 2)
   --> /build/cargo-vendor-dir/move-prover-0.1.0/src/cli.rs:774:33
    |
774 |                   template_bytes: include_bytes!(
    |  _________________________________^
775 | |                     "../../../../aptos-move/framework/src/aptos-natives.bpl"
776 | |                 )
    | |_________________^
    |
    = note: this error originates in the macro `include_bytes` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `move-prover` (lib) due to 1 previous error

And that is definitely an error in that move-prover crate. move-prover is part of the git repo of aptos-core but isn't part of any workspace. This causes cargo vendor to vendor it by itself, but the include_bytes! directive is written in a way that expects the move-prover crate to be part of that aptos-core git repo. Running cargo build without the vendoring step builds move-prover as part of the original repo and thus works by accident.

This isn't a bug with Nix. It's just a little curious behaviour in cargo vendor.

Here's the patch I've used to get to that error: fix-nix-files.patch.txt

l-monninger commented 4 weeks ago

Yeah, that's the error! Thanks for taking a closer look!

We had hypothesized that this was because of a source filter being applied to vendored dependencies. I believe @andyjsbell had actually found the particular spot in Crane where that appeared to occur. But, if this is down to Cargo Vendor behavior, that would make sense and prevent us running that fools errand.

I'm going to leave open until I have time to confirm.

NixOS / nixpkgs