cargo add selects versions of native dependencies which are not compatible with transitive packages

udoprog commented 1 year ago

Problem

Originally reported in diesel-rs/diesel#3587

The issue here is that diesel depends on libsqlite3-sys, which is a native dependency and declares that it links to sqlite3. If the crate being added to gets a direct dependency to libsqlite3-sys (something which is needed to activate features in it that are not re-exported by diesel) this can cause cargo add to potentially select an incompatible version.

Steps

If you run cargo add libsqlite3-sys on a manifest with the following:

[dependencies]
diesel = { version = "2.0.3", features = ["sqlite", "chrono"] }

It results in the following error:

    Updating crates.io index
      Adding libsqlite3-sys v0.26.0 to dependencies.
             Features:
             + min_sqlite_version_3_14_0
             + pkg-config
             + vcpkg
             - bindgen
             - buildtime_bindgen
             - bundled
             - bundled-sqlcipher
             - bundled-sqlcipher-vendored-openssl
             - bundled-windows
             - bundled_bindings
             - cc
             - in_gecko
             - openssl-sys
             - preupdate_hook
             - session
             - sqlcipher
             - unlock_notify
             - wasm32-wasi-vfs
             - winsqlite3
             - with-asan
    Blocking waiting for file lock on package cache
error: failed to select a version for `libsqlite3-sys`.
    ... required by package `diesel v2.0.3`
    ... which satisfies dependency `diesel = "~2.0.0"` of package `diesel_migrations v2.0.0`
    ... which satisfies dependency `diesel_migrations = "^2.0.0"` of package `oxidize v0.0.0 (D:\Repo\*snip*)`
versions that meet the requirements `>=0.17.2, <0.26.0` are: 0.25.2, 0.25.1, 0.25.0, 0.24.2, 0.24.1, 0.24.0, 0.23.2, 0.23.1, 0.23.0, 0.22.2, 0.22.1, 0.22.0, 0.20.1, 0.20.0, 0.18.0, 0.17.3, 0.17.2

the package `libsqlite3-sys` links to the native library `sqlite3`, but it conflicts with a previous package which links to `sqlite3` as well:
package `libsqlite3-sys v0.26.0`
    ... which satisfies dependency `libsqlite3-sys = "^0.26.0"` of package `oxidize v0.0.0 (D:\Repo\*snip*)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the 
links ='libsqlite3-sys' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `libsqlite3-sys` which could resolve this conflict

Note: The same issue applies to cargo upgrade, which comes from the cargo-edit project for which this issue was originally reported.

Possible Solution(s)

Cargo could opt to only upgrade to versions of libsqlite3-sys which are compatible. That means it would have to analyze linking restrictions when selecting a new version.

Cargo could also opt to allow the activation of transitive features. I'm sure that's been discussed somewhere but I can't find it right now.

Note that there's also the option for the transitive crate to re-export the desired feature, but in the case of diesel the maintainer is unwilling to do so.

Notes

No response

Version

No response

ehuss commented 1 year ago

cargo upgrade is not a command provided by cargo. It is a third-party command. Can you move this issue over to https://github.com/killercup/cargo-edit?

udoprog commented 1 year ago

Ah, that makes sense. I believed cargo upgrade was added here at the same time as cargo add. Sorry!

It seems that cargo add suffers from a similar issue of selecting an incompatible version, so perhaps this issue could be about that one? I don't know exactly how cargo upgrade is implemented, but I suspect fixing one might provide for the plumbing to fix the other.

D:\Repo\*snip*〉cargo add libsqlite3-sys                                                                                                                                                04/10/2023 04:12:39 AM
    Updating crates.io index
      Adding libsqlite3-sys v0.26.0 to dependencies.
             Features:
             + min_sqlite_version_3_14_0
             + pkg-config
             + vcpkg
             - bindgen
             - buildtime_bindgen
             - bundled
             - bundled-sqlcipher
             - bundled-sqlcipher-vendored-openssl
             - bundled-windows
             - bundled_bindings
             - cc
             - in_gecko
             - openssl-sys
             - preupdate_hook
             - session
             - sqlcipher
             - unlock_notify
             - wasm32-wasi-vfs
             - winsqlite3
             - with-asan
    Blocking waiting for file lock on package cache
error: failed to select a version for `libsqlite3-sys`.
    ... required by package `diesel v2.0.3`
    ... which satisfies dependency `diesel = "~2.0.0"` of package `diesel_migrations v2.0.0`
    ... which satisfies dependency `diesel_migrations = "^2.0.0"` of package `oxidize v0.0.0 (D:\Repo\*snip*)`
versions that meet the requirements `>=0.17.2, <0.26.0` are: 0.25.2, 0.25.1, 0.25.0, 0.24.2, 0.24.1, 0.24.0, 0.23.2, 0.23.1, 0.23.0, 0.22.2, 0.22.1, 0.22.0, 0.20.1, 0.20.0, 0.18.0, 0.17.3, 0.17.2

the package `libsqlite3-sys` links to the native library `sqlite3`, but it conflicts with a previous package which links to `sqlite3` as well:
package `libsqlite3-sys v0.26.0`
    ... which satisfies dependency `libsqlite3-sys = "^0.26.0"` of package `oxidize v0.0.0 (D:\Repo\*snip*)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the 
links ='libsqlite3-sys' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `libsqlite3-sys` which could resolve this conflict

epage commented 1 year ago

This somewhat reminds me of the problem of public/private dependencies.

One overall complication is how to decide to walk the dependency tree. I feel like we'd need to walk the dependency tree, enumerating links, for all features and platforms. Something that simplifies this is that the lack of a feature cannot add a dependency. Should we also take into account workspace members that depend on the current crate? What about adding a crate that depends on a crate with a links, like the rust wrapper to a -sys crate? I'm not sure we can do a good job calculating what version we should use in those cases.

rust-lang / cargo