buildRustPackage: found duplicate version of package

[ljli]

I wanted to upgrade rustup to 1.6 now that we have a recent rust compiler again thanks to #30088. Unfortunately I got an error, the log tail is below. Rustup uses a [replace] section for mio in its Cargo.toml, so that is probably the cause for the error. The error output itself is from cargo-vendor. I would report it there but I'm not sure why it can't handle this, maybe there is some good reason and we should use it in a different way. There shouldn't be any trouble to reproduce this, updating the hash and version for the rustup derivation should do. I suspect this is a general problem with cargo packages that use [replace], hence the issue name.

$ nix-build '<nixpkgs-work>' -A rustup --no-out-link
...
 Downloading aho-corasick v0.5.3
 Downloading lzma-sys v0.1.8
 Downloading foreign-types v0.2.0
 Downloading gcc v0.3.53
error: failed to sync

Caused by:
  found duplicate version of package `mio v0.6.10` vendored from two sources:

    source 1: https://github.com/carllerche/mio#6871f83a
    source 2: registry https://github.com/rust-lang/crates.io-index
builder for ‘/nix/store/ycmixz6k8djya309dl67x4px97qr595c-rustup-1.6.0-vendor.drv’ failed with exit code 101
cannot build derivation ‘/nix/store/sq4ci1azfqg1jnmsj6zdjcjaj89cdbjk-rustup-1.6.0.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/sq4ci1azfqg1jnmsj6zdjcjaj89cdbjk-rustup-1.6.0.drv’ failed

[wizeman]

cc @zimbatm

[zimbatm]

I did the packaging but actually don't have much insight on how the internals are working. The cargo-vendor author might be able to tell us why the [replace] directive is not overriding the crates.io entry.

[aakropotkin]

Hey I have run into this issue trying to build a package for "spotifyd". I get duplicate versions of 'rust-crypto', which is the only [replace] directive in the project.

Did you ever find a solution?

[ljli]

FTR, cargo-vendor has a flag (--no-merge-sources) now which solves the root problem. If buildRustPackage gets adapted, crates like spotifyd should work.

[bobvanderlinden]

I was also working on a package for spotifyd (see https://github.com/NixOS/nixpkgs/pull/47172), but just like @BadDecisionsAlex I ran into this issue. @ljli I couldn't find the no-merge-sources in cargo-vendor. I tried adding --no-merge-sources to the cargoBuildFlags, but that still resulted in the same error. Any ideas?

Also, I was able to build spotifyd inside nix-shell:

nix-shell -p cargo -p alsaLib -p libpulseaudio -p pkgconfig -p openssl -p dbus
cargo build --release --features pulseaudio_backend

[ljli]

The flag --no-merge-sources is available in the current cargo-vendor master branch. So first of all we need to update it in nixpkgs. The place to pass the flag would then be here: https://github.com/NixOS/nixpkgs/blob/fcde178ed5f76626d57b3b02848f2fedf5fd9928/pkgs/build-support/rust/fetchcargo.nix#L45 I'll try to have a look tomorrow.

[ljli]

My first attempt to hook up --no-merge-sources is here: https://github.com/ljli/nixpkgs/tree/cargo-vendor-multi-sources

I think something along those lines should work, it is based on staging because of #46362. Unfortunately I hit the following now:

$ nix build nixpkgs-work.spotifyd
error: path '/nix/store/kyvsqxp7cz3f0dhf1l7ly1vw9208wn88-spotifyd-0.2.2-vendor' is not valid

I have no clue why, maybe this is a local problem so you are welcome to try the branch and/or pick it up from there, otherwise I'll investigate after staging got merged.

[boomshroom]

Could you perhaps submit a pull request for that? I'm running into the same problem.

[zimbatm]

A bunch of things have changed on the rust packaging side which probably fix this issue. Ping me if I should re-open.

[bqv]

@zimbatm I think this issue is still quite relevant. See:

rustPlatform.buildRustPackage {
            pname = "weechat-discord";
            version = "test";

            src = fetchFromGitHub {
              owner = "terminal-discord";
              repo = "weechat-discord";
              rev = "e09bac671b14c389f9909d8393ab76a575c5df73";
              sha256 = "0n1681vn0w2a4zjfmaimbbjsfpc5aflp9rr5w7j3zh1mp87wnxdj";
            };

            cargoSha256 = "0000000000000000000000000000000000000000000000000000";

            buildInputs = [
            ];
          }

Which for me, fails with:

  (...)
  Downloaded winapi-i686-pc-windows-gnu v0.4.0
error: failed to sync

Caused by:
  found duplicate version of package `parking_lot v0.9.0` vendored from two sources:

        source 1: https://github.com/terminal-discord/parking_lot?rev=046a171#046a1714
        source 2: registry `https://github.com/rust-lang/crates.io-index`
Traceback (most recent call last):
  File "/nix/store/14qgqb8avba5fbpsdwmnys49h2k8ljmk-cargo-vendor-normalise/bin/.cargo-vendor-normalise-wrapped", line 42, in <module>
    main()
  File "/nix/store/14qgqb8avba5fbpsdwmnys49h2k8ljmk-cargo-vendor-normalise/bin/.cargo-vendor-normalise-wrapped", line 17, in main
    assert list(data.keys()) == ["source"]
AssertionError
builder for '/nix/store/ccwiz2jm9kvfgc9qk3gddxpy7m6c1par-weechat-discord-beta-vendor.drv' failed with exit code 1
cannot build derivation '/nix/store/gg7fwmp6yr5xh83w4idxla65nxa2ccqp-weechat-discord-beta.drv': 1 dependencies couldn't be built
error: build of '/nix/store/gg7fwmp6yr5xh83w4idxla65nxa2ccqp-weechat-discord-beta.drv' failed

[zimbatm]

On what version of nixpkgs did you experience the issue? What are the instructions to reproduce this error independently?

[bqv]

Unstable and/or master, and simply trying to build that expression suffices to create that error.

Here is it wrapped in a shell script for your convenience: https://termbin.com/fo72

[FlorianFranzen]

I ran into the same error while building a project with a direct dependency that is also a dependency of another direct dependency but from a different source. The resulting Cargo.lock then lists both source separately:

https://github.com/paritytech/polkadot/blob/master/Cargo.lock#L6694-L6702

The package builds fine in a nix-shell, but fails in the buildRustPackage environment.

[bqv]

Hit this again trying to build servo. Has anyone got any ideas for this?

[zimbatm]

Is there a code somewhere where I can just run nix-build on it to reproduce the error?

[bqv]

I posted a shell script a month ago, assuming you didn't try it (the link is now dead). Here is the exact same expression fleshed out to a self-contained default.nix and gisted. It reliably reproduces the error:

https://gist.github.com/bqv/3209cb2115075797631b8cd33775a40a

Suffice to say, trying to buildRustPackage any of the three projects we've mentioned will result in this failure.

[zimbatm]

here is a minimally-reproducible version:

{ pkgs ?
  import <nixpkgs> {
    config = {};
    overlays = [];
  }
}:

pkgs.rustPlatform.buildRustPackage {
  pname = "weechat-discord";
  version = "test";

  src = pkgs.fetchFromGitHub {
    owner = "terminal-discord";
    repo = "weechat-discord";
    rev = "e09bac671b14c389f9909d8393ab76a575c5df73";
    sha256 = "0n1681vn0w2a4zjfmaimbbjsfpc5aflp9rr5w7j3zh1mp87wnxdj";
  };

  cargoSha256 = "0000000000000000000000000000000000000000000000000000";
}

[bhipple]

If you run this:

$ git clone https://github.com/terminal-discord/weechat-discord
$ cd weechat-discord
$ cargo vendor
...
Caused by:
  found duplicate version of package `parking_lot v0.9.0` vendored from two sources:

        source 1: https://github.com/terminal-discord/parking_lot?rev=046a171#046a1714
        source 2: registry `https://github.com/rust-lang/crates.io-index`

You will get the same error message, since their Cargo.lock file does indeed appear to specify the parking_lot dependency twice from two different locations: https://github.com/terminal-discord/weechat-discord/blob/master/Cargo.lock#L877-L895

The nix build relies on cargo vendor to succeed, so if Cargo can't handle the project's dependencies, nix won't be able to either. My guess is that one of their transitive dependencies relies on parking_lot but doesn't pin it to the same git commit?

Interestingly, cargo-vendor is now deprecated, since it was moved into cargo itself, but the moved version doesn't support the mentioned --no-merge-sources:

$ cargo vendor --no-merge-sources
error: the crates.io `cargo vendor` command has now been merged into Cargo itself
and does not support the flag `--no-merge-sources` currently; to continue using the flag you
can execute `cargo-vendor vendor ...`, and if you would like to see this flag
supported in Cargo itself please feel free to file an issue at
https://github.com/rust-lang/cargo/issues/new

[bqv]

Correct, this is the case for all three projects. Cargo build succeeds fine on all three however, so we shouldn't be blaming this on cargo

[bhipple]

Unfortunately, buildRustPackage depends on cargo vendor to fetch the src dependencies into the build sandbox prior to compilation, so if cargo isn't able to vendor the project we're in trouble. It looks like @ljli implemented some support for this in cargo-vendor, but it appears not to have been lifted-and-shifted over to cargo vendor :(

[offlinehacker]

Same issue when trying to build https://github.com/cloud-hypervisor/cloud-hypervisor Any new information/ideas how to solve this issue?

[FlorianFranzen]

To ensure that cargo downloads all the dependencies in a reproducible manner, cargo-vendor is used as an in between step (which is also why you need to specify a cargoSha256 in the derivation).

However cargo-vendor is currently unable to vendor dependencies if the same version of crate is needed but from different source (e.g. crates.io and github.com). If your build fails with aforementioned error, that just means that the project you are trying to build does not support vendoring (given cargo-vendor's limitations/bugs atm), which in many cases is something that can be patched by using the same source throughout. Just remember to use the cargoPatches attribute instead of patches.

Here is for example what I do to fix vendoring for Parity polkadot.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[bqv]

still not solved.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[YellowOnion]

Should this be reported upstream to cargo?

Is there any work around, I want to build lapce.

[teto]

got a similar issue when trying to update neovide, sadly I dont know enough about rust to help

       >   Downloaded ndk-glue v0.3.0
       >   Downloaded crossbeam-utils v0.8.5
       >   Downloaded clipboard-win v2.2.0
       > error: failed to sync
       >
       > Caused by:
       >   found duplicate version of package `winit v0.24.0` vendored from two sources:
       >
       >    source 1: https://github.com/Kethku/winit?branch=new-keyboard-all#fc49a506
       >     source 2: https://github.com/neovide/winit?branch=new-keyboard-all#fc49a506
       > Traceback (most recent call last):
       >   File "/nix/store/qhh6v7damlvpizwyvac47165mi0hf8li-cargo-vendor-normalise/bin/.cargo-vendor-normalise-wrapped", line 42, in <module>
       >     main()
       >   File "/nix/store/qhh6v7damlvpizwyvac47165mi0hf8li-cargo-vendor-normalise/bin/.cargo-vendor-normalise-wrapped", line 17, in main
       >     assert list(data.keys()) == ["source"]
       > AssertionError
       For full logs, run 'nix log /nix/store/3ns0cqaknvnka364dmzh60ycqs3h0ggm-neovide-unstable-2021-10-09-vendor.tar.gz.drv'.

[diogox]

Is there a workaround for this?

[YellowOnion]

There's a PR open to fix this https://github.com/rust-lang/cargo/pull/10344, maintainers of buildRustPackage should read and make sure it works for this use case.

[YellowOnion]

cc @danieldk @Ericson2314 @Mic92 @LnL7 @zowoq sorry to bother, just trying to figure out who manages buildRustPackage or can help audit the above mentioned PR.

[ruuda]

I ran into this issue, and in my case it was possible to work around / fix it in the upstream Cargo package. In my case, the problem was a patched dependency through [patch.crates-io] in Cargo.toml, but that dependency was coming from a Git repository with many interdependent crates, and I did not patch all of them. Adding an entry to the [patch.crates-io] section for these transitive dependencies until cargo vendor stopped complaining fixed the issue for me.

[kvtb]

Can we get rid of broken cargo vendor for the purpose of buildRustPackage?

We could reconstruct repository at $HOME/.cargo instead of ./vendor directory (akin to $HOME/.m2 is reconstructed for java projects, which lack any maven vendor).

[kvtb]

Using cargo vendor has another issue: it tries to build every package under ./vendor/, even those which are not suppose to built.

For example, if you package TiKV, even after working around 3 duplicates, you'll hit at build failure of ./vendor/libtitan_sys-0.0.1 - the package which should not be built as an independent package at all.

[kotatsuyaki]

I ran into this issue when trying to wgsl-analyzer. In their ide package, they use multiple versions of the naga library on purpose. Working around the issue seems to require patching a substantial amount of Rust code (to drop support for all versions of naga except for one), which is not sustainable and results in lost of functionality.

With the cancellation of rust-lang/cargo#10344 due to potential syntactic conflict with another Rust RFC, the problem is unlikely to be solved upstream soon. I support @kvtb's idea of migrating away from cargo vendor for buildRustPackage.

[junjihashimoto]

I reimplemented cargo vendor --no-merge-sources for cargo. https://github.com/rust-lang/cargo/pull/13271

[junjihashimoto]

Actually, I don't know if it will be merged or not. This issue seems to be resolved by applying a patch to cargo.

[junjihashimoto]

The cargo's build itself works with cargo vendor, so we can use patched cargo.