search

rust-lang / wg-cargo-std-aware

Repo for working on "std aware cargo"
133 stars 8 forks source link

`Cargo.lock` and dependency resolution #12

Closed ehuss closed 1 year ago

ehuss commented 4 years ago

This issue is for working through the implementation issues with locked dependencies.

It is almost certain that the standard library will need to be built with the exact same dependencies used in the release. That is, dependencies like libc will not be allowed to "float" to the most recent release. All dependencies will also be built separately from the user's project, so if the user has a dependency on libc, it will be built separately from the libc for the standard library (which may be the same or different version). Metadata tricks are used to link these properly.

The rust-src rustup component includes the standard library sources, but does not include dependencies. It includes the Cargo.lock file, from which we can infer the dependency versions.

I'm not certain at this time what problems this might present.

alexcrichton commented 4 years ago

One other reason I think we need to lock dependencies is for determinism. Or well I think that we should provide a way to have deterministic builds of the standard library (no floating deps over time) and unless we store more information in project lock files (which I'm not sure is feasible) then using the rust lock file would solve this issue

SimonSapin commented 4 years ago

All dependencies will also be built separately from the user's project, so if the user has a dependency on libc, it will be built separately from the libc for the standard library

This could be represented in Cargo’s resolution as a separate "source", for example std+registry+https://github.com/rust-lang/crates.io-index instead of registry+https://github.com/rust-lang/crates.io-index.

Ericson2314 commented 4 years ago

Hmm there's two issues at play: making the depedencies not "float" and making them not conflict with the users dependencies. For the latter, we can and should use private dependencies. They are *exactly" what we need, and will also help us catch the standard library leaking implementation details. For the former one can just pin exact versions from the stdlib Cargo.toml? Remember with different features/platform combinations, the exact dependencies of std and friends can change.

Eh2406 commented 4 years ago

Um... That is not what private dependencies do according to the current RFC. Private dependencies are just as locked to the rest of the dependency tree as the current "unspecified" dependencies. IE only one global use of each links and no two semver compatible versions.

Ericson2314 commented 4 years ago

no two semver compatible versions.

Well that is silly. Private dependencies should never unify (i.e. even if we use the same libc it should show up as if we used two different ones so we cannot accidentally mix them together.) Therefore maybe Cargo can prefer to user fewer versions, but if std depends on an exact version and something else another exact version, the build plan should still go through.

Eh2406 commented 4 years ago

That is an interesting thought, probably worth discussing a new RFC for that, but for now that is not what "private dependencies" does. So we will need to do something special for std.

Ericson2314 commented 4 years ago

Well for the MVP we don't. I personally think this is a minor tweak to that RFC so I'll comment in that issue, and see what people say.

ehuss commented 1 year ago

I'm going to close since I don't expect any changes regarding this for the foreseeable future.

build-std currently uses the Cargo.lock from the rust repo to ensure the dependencies are locked, and uses a separate resolver to ensure that the dependencies are kept independent and private from the user's dependencies.

It might be nice if that wasn't needed, but I don't think there is going to be a different solution, such as something that allows duplicate semver-compatible dependencies. That isn't on our radar, and would be a whole initiative on its own.

Alternatively, I am skeptical that allowing semver-compatible updates of dependencies is feasible since those dependencies would need to go under a much greater level of scrutiny than they currently do (else a breaking change could break all cargo users instantly). Some dependencies like compiler_builtins could do a semver-major version bump for every release to avoid that, but that would only be a partial solution.

If other issues require a change in how that works, then any changes will need to motivated and branched from there.