"cargo install" apparently ignores "Cargo.lock" as opposed to "cargo build"

[df5602]

I've encountered a strange issue today.

Background: The percent-encoding crate recently made a new major release. Knowing that stuff could break I held off on updating the crate because I haven't yet had time to check what changed.

Problem When I compile my binary crate using cargo build (with and without --release) everything builds fine. However, if I compile my binary using cargo install --path . -f, compilation fails with an error related to percent-encoding (the breaking change warranting the major version bump I assume..). In the output I see that it compiled percent-encoding v2.0.0 instead of percent-encoding v1.0.1 as specified in Cargo.lock. (Note: The Cargo.toml doesn't specify a specific version.)

The crate is the following, if you want to try at home: https://github.com/df5602/bingers

Steps

1. $ cargo clean
2. $ cargo build

   ...
   Compiling percent-encoding v1.0.1
   ...
   Finished dev [unoptimized + debuginfo] target(s) in 48.32s

3. $ cargo build --release

   ...
   Compiling percent-encoding v1.0.1
   ...
   Finished release [optimized] target(s) in 1m 25s

4. $ cargo install --path . -f

   
   Installing bingers v0.1.0 (/path/to/bingers)
   Updating crates.io index
   Compiling percent-encoding v2.0.0
   Compiling bingers v0.1.0 (/path/to/bingers)
   error[E0432]: unresolved import `percent_encoding::QUERY_ENCODE_SET`
   --> src/tvmaze_api.rs:10:45
   |
   10 | use percent_encoding::{utf8_percent_encode, QUERY_ENCODE_SET};
   |                                             ^^^^^^^^^^^^^^^^ no `QUERY_ENCODE_SET` in the root

error: aborting due to previous error

I can work around it by specifying an explicit version in `Cargo.toml`, but the observed behaviour was surprising to me...

**Notes**

Output of `cargo version`:

`cargo 1.36.0 (c4fcfb725 2019-05-15)`

OS: Mac

[ehuss]

This has been changed in 1.37 (beta). Starting in that version the --locked flag can be used to force cargo to use the Cargo.lock file.

The intent is that cargo install will behave the same whether it is from a registry or a local path.

We discussed this a bit (https://github.com/rust-lang/cargo/pull/6840#pullrequestreview-226227411 is most relevant). Ideally Cargo would issue a message with a hint to try --locked, but I'm not so confident that would work well. I think since it can result in any error, it wouldn't be able to know if --locked would actually fix it, so I think it would be misleading too often.

[df5602]

1. I'm using cargo 1.36 (stable) which shouldn't show that behaviour yet, right? (Running cargo install with --locked worked, though...)

2. Reading up on the linked issue: if I understand correctly, going forward, the behaviour will be different between cargo build and cargo install regarding dependency resolution (at least for binary crates)? If yes, I would consider that as very surprising! (So, what's the Cargo.lock for? Basically like this, you compile, run your tests and then cargo install possibly picks different dependencies?)

I know, I should probably fix my Cargo.toml and specify proper versions instead of using "*". But until now, it never really mattered for binary crates. Updating dependencies was always a manual process, so that you can deliberately audit the changes if you wish to do so (for me it was basically running cargo outdated, check the listed crates and look for the changelog, and eventually running cargo update -p xyz). If that behaviour changes, I would expect at least a warning à la "Warning: Ignoring lock file!" and probably a PSA in the Rust release notes.

[ehuss]

1. Running `cargo install` with `--locked` worked, though...

Ah, yea, in that sense there is no change in behavior. I mis-remembered how it used to work. The present stable behavior for install --path is that Cargo.lock is ignored by default, and --locked can be used to honor it. That hasn't changed. (What did change is installing from a registry, which didn't support lock files at all.)

It's difficult to balance the defaults here. I think it would be confusing if cargo used Cargo.lock differently based on the source. And for now we don't want to default to using Cargo.lock for installing from a registry. I think that's up for debate on changing it, but it was the conservative choice we made for now. If we ever decide to change it, we would probably want to add a flag to override it, and that adds more complexity.

[df5602]

Ah ok, so I guess it was the first time I ran into this because I deliberately held off on updating a dependency (due to breaking changes) which happens quite rarely...

It's good to know that this is the default behaviour, but I think this should be a lot better documented, because it's highly unintuitive and surprising: Until now, my mental model of cargo install --path was basically "run cargo build --release and copy the binary somewhere". But apparently that's wrong, it should be "compile the binary slightly differently than cargo build --release and copy the binary somewhere".

I think it's the wrong default for binary crates because it means that you're deploying different binaries than you're testing! (In my opinion it would be ok, if cargo build would ignore lockfiles by default and you could explicitly opt-in to using lockfiles. But currently the default is otherwise.)

[ehuss]

I think this should be a lot better documented

There's a fairly lengthy paragraph on https://doc.rust-lang.org/nightly/cargo/commands/cargo-install.html that explains how cargo install handles Cargo.lock (this is new for 1.37, so it hasn't hit stable, yet). If there's any part that could be clarified let us know, we're always eager to improve the documentation.

[df5602]

I assume this will also be part of the cargo install --help page? I think the behaviour is described very clearly in the updated help page.

Given that the described behaviour is apparently expected behaviour, I think we can close this issue?

[MaulingMonkey]

https://doc.rust-lang.org/nightly/cargo/commands/cargo-install.html has some misleading documentation under "Manifest Options" for --frozen / --locked:

The --frozen flag also prevents Cargo from attempting to access the network to determine if it is out-of-date.

This is untrue for cargo install. In fact, I don't think this can ever succeed for install unless the crate has no dependencies. Maybe not even then.

These may be used in environments where you want to assert that the Cargo.lock file is up-to-date (such as a CI build) or want to avoid network access.

This is inaccurate for cargo install. If anything, you'd use these flags to use an "out-of-date" lock file - in the sense that all the documentation suggesting cargo install xyz will be using different versions.

If the lock file is missing, or it needs to be updated ...

This is technically accurate, but the way it implies it might not need to be updated is kinda confusing, as which crates are used will always be updated without these flags - for the install build at least - regardless of if they "need to be" or not.

If this documentation is automatically shared with other commands, it might at least be worth breifly mentioning cargo install being a special case. Maybe something like:

Unlike cargo build, cargo install will ignore Cargo.lock files completely without --locked."

?

Ideally Cargo would issue a message with a hint to try --locked, but I'm not so confident that would work well.

Instead of trying to guess when an error can be fixed by --locked, it might be useful to instead warn if Cargo.lock is present, and specifies different versions than are being used, since that's just begging for version confusion and building everything twice needlessly. --locked is one possible fix, but cargo update is another - since presumably most docs suggest cargo install xyz rather than cargo install xyz --locked?

[MaulingMonkey]

(I also suspect I landed on the "Manifest Options" section by searching the cargo install docs page for "--frozen", which caused me to skip over the earlier/better explaination of locking behavior entirely.)

[ehuss]

I assume this will also be part of the cargo install --help page?

Not for now, unfortunately we don't share the man pages with the help text. The long-term intent is to make it available in some fashion.

If this documentation is automatically shared

Yes, it is shared. We can probably make it special-cased for install, but that would take a bit of restructuring.

[df5602]

In that case I take back what I said earlier: I don't think I'd have the idea to look-up a description of a command online, if said command also has an extensive man page, so I still consider it undocumented behaviour...

I also agree with @MaulingMonkey: If this is the desired behaviour, then it should at least generate warnings, something along those lines (modulo wordsmithing):

Warning: Ignoring `Cargo.lock` file!
Detecting version mismatch:
    | Specified in `Cargo.lock` | Used in build
------------------------------------------------
foo |                    v1.0.1 |        v1.0.3

... possibly followed by a few suggestions on how to deal with the situation (i.e. use --locked, run cargo update, etc.)

(Although I don't know the inner workings of cargo, so I can't judge how much work that would be...)

[Nemo157]

(EDIT: Moved to https://github.com/rust-lang/cargo/issues/7495)

[matklad]

I've hit this in rust-analyzer a couple of times, and I would appreciate the warning. How does this interact with publish-lockfile? Will adding publish-lockfile = true to Cargo.toml make Cargo to use lockfile when installing? Or is the publish-lockfile feature effectively a no-op?

[ehuss]

How does this interact with publish-lockfile?

publish-lockfile has been removed and is a no-op now. Cargo now automatically publishes the lock file if there are any binaries in the package. cargo install always ignores the Cargo.lock unless you specify --locked.

[nixpulvis]

Let me see if I'm understanding this correctly, because lockfiles have been a pet-peeve of mine in other ecosystems (looking at you NPM).

• cargo build will respect the Cargo.lock file installing only dependencies that are known to work, since those are the versions everyone is building with
• cargo install will ignore the Cargo.lock file, installing the latest matching versions with respect to the Cargo.toml file, presumably because there are security or other important patching an installing user may want

This seems wrong, since it seems like this whole setup assumes crates are following semantic versioning, strictly. As far as I'm concerned cargo build is to make as cargo install is to make install (as things are typically implemented). If we have a lockfile, both operations should respect it. How else will I know that people's builds (and installs) will work!?

As the author of Rust software, I want to use cargo update to try to update my dependencies, but I might hit problems, and they'll need to be resolved before I publish my updated lockfile. Of course, an ideal release of software is tested in some sense!

On the other hand I may find myself using only dependencies who follow "semantic versioning" effectively, and thus may decide to offer users automatic updates for dependencies. In this case I'd need to lock non-conforming dependencies explicitly in the Cargo.toml moving forward.

Now, since we do allow published crates to be installed remotely, and some of these crates will become stale, it makes sense to allow new users of software correctly following semantic versioning to automatically get the updates. But this should be an opt-in process, since not all software follows semantic versioning (or is generally capable of handling automatic updates to dependencies).

I might be missing a lot of details for why this next idea won't work, but in an automatically updating setup (like one following semver), I think I'd expect to not have a Cargo.lock file at all.

[ehuss]

I personally would be fine with switching the default cargo install behavior, as I feel reliable builds are more important than possibly getting "fixed" dependencies. But there has been resistance to this in the past. I think I have mitigated some of the concern by issuing warnings on yanked dependencies. But presuming few use --locked, we don't really know what the impact of that will be. The warning isn't very helpful, since most users will have no idea what it means or control over updating those dependencies, so I imagine it has a risk to be confusing at the least.

There's also a bit of risk that this would change current behavior, which some projects may be relying upon in some way. I'm not sure how things like dh-cargo would be impacted, or what they would prefer.

[ojeda]

It is surprising, indeed, and a security concern too: consider projects/apps that may just provide source tarballs or a repo somewhere, specially small ones within the Linux ecosystem.

End-users will download a package and then run cargo install. They may or may not know about Rust/Cargo. Now:

• If they are lucky, the binary works as intended.
• If they are not so lucky, it doesn't.
• And if they are really unlucky, arbitrary nasty code just got executed.

And all this happens even if they have verified the checksums/signatures of their downloads or their checked out signed Git tags!

I would say:

• At least both build and install should behave the same in all respects (except actually installing, of course). I believe this is what users of traditional build systems expect.
• Both commands should default to failure if Cargo.lock needs to be modified for any reason. For binaries, they should also fail if it has to be created. This makes the behavior more consistent by default (i.e. whether Cargo.lock is there does not change behavior of build), makes the act of updating the dependencies explicit, whether via update or a shorthand, and ensures source releases work as intended.

[anguslees]

(This is another example of "I don't want anything to change, but I also want new changes". It's surprising how often this obviously (when expressed in these simple terms) irreconcilable conflict arises :P )

[NickeZ]

As a "simple user" I find this behavior completely unexpected and therefore breaks the most important rule of all rules when it comes to software development: https://en.wikipedia.org/wiki/Principle_of_least_astonishment

I have no idea if you have the power to change this, but I would change it even if it breaks backwards compatibility. AFAIU cargo install will strictly work more often than currently by changing the default behavior to enable --locked. Are there any cases where cargo install would not work if it used --locked?

Edit: Also I looked like an idiot reporting this issue in the "Alacritty" repository when this bug is in Cargo. And this is like the second most important rule in UX. Never make your users look stupid.

[nixpulvis]

@NickeZ I think the "case" that won't work is automatic (potentially security related) updates for downstream installers. However, as I've mentioned above, I believe that should be handled differently. Also, don't worry too much about looking like an idiot. This behavior surprised me too, and having some discussion on it in a now closed issue doesn't bother us.

[ghost]

Our team got hit by this today and we found this behavior to be very surprising. Consistency is the key here, defaulting to use lock file for something and not for others are very inconsistent / surprising / confusing.

[MartinKavik]

If I'm not wrong, this behavior broke wasm-pack installation and CI pipelines for multiple people - more info: https://github.com/rustwasm/wasm-pack/issues/818#issuecomment-602274678

[jhunt]

Ran into this as well this morning, when I switched up Docker build hosts and lost my old copy of the rust base image (I know, I know -- shame on me for not pinning the tag). The dichotomy seems surprising to me, and I'm not sure how I feel about the argument for "cargo install will pick up the security patches" - are people regularly cargo install-ing their upstream binaries over and over again? I rarely do one-off re-installs to get updates. Either the package ecosystem provides a mass-method (apt upgrade and friends) or I use something else with a high churn-rate (rebuilding Docker images nightly/weekly/etc.)

While a hint about the absence of --locked in the presence of Cargo.lock might be lost on some users (arguably not those who are cargo install-ing their software), it would definitely cut down on the problem I'm seeing: Docker builds breaking inexplicably.

My problem started when a tried to compile something with Diesel + UUID - cargo build (honoring the lock file) was happily installing 0.7.4 of uuid, and (on my local dev machine, outside of Docker) everything was fine.

When I went to build the Docker images to crank out a quick demo environment for a colleague, I ran into a whole bunch of errors about uuid::UUID not implementing the trait bounds needed for Diesel ORM bits (Insertable and co.). Google turned up a (seemingly) far-fetched root cause: multiple versions of UUID. I blew away my local compiled cache and did a new cargo build locally - just the one version, v0.7.4. When I looked at the failing Docker build output, however, I saw that cargo (via an unlocked install) was indeed installing both v0.7.4 (my dep) and v0.8.1 (no idea where that came from).

I did some more digging, found a thing called cargo tree that shows a dependencies graph, but still couldn't trace the source of the new dep. On a whim, I decided to change the Docker image to use cargo build instead of cargo install and it worked.

That, in turn, lead to another jaunt through the googles looking for why cargo install --path . and cargo build do different things.

(Apologies for the long-winded narrative, but this has eaten about 2h out of my day for a task with an expectation of "30s to kick off; go get a cup of tea check back later" levels of required focus).

What I personally would have liked to have seen:

1. A warning at the top of the log, at the beginning of cargo install's execution, saying something like "Hey there, I see you have a Cargo.lock file (at ./Cargo.lock), but are not running with --locked. This might break some things [and here's why]."
2. The same warning, in the event that the installation fails.

Having the warning at both ends of the log means that (a) I see it when it breaks and there's tons of compiler errors between start-of-exec and the final exit, and (b) I see it if I run cargo install manually, before I get too far into downloading and compiling the several hundred crates that make up my project.

[RalfJung]

A warning at the top of the log, at the beginning of cargo build's execution, saying something like

Did you mean cargo install's execution?

[jhunt]

Indeed I did. Oops! Corrected.

[ehuss]

are people regularly cargo install-ing their upstream binaries over and over again?

It's not exactly about regularly installing, it's more about regularly publishing. Let's say there is a security update in a popular library. There could be thousands of binary projects relying on it directly or indirectly. If Cargo used the original Cargo.lock, all of those binary projects would need to publish an update, otherwise all users will get the old version.

I realize it is awkward and confusing, but switching the default has its own drawbacks. It's not really clear which is best.

Implementing the hint is on my todo list, will hopefully get to it at some point.

[ojeda]

If Cargo used the original Cargo.lock, all of those binary projects would need to publish an update, otherwise all users will get the old version.

There are several problems with that approach:

• End users that already installed their tools won't get the update (what @jhunt meant, I think), because virtually all end users are used to check if there is a new release/tag/announcement, and if not, stop right there.
• End users that installed in different days may have an app that differs in behavior from someone else's build, while their --version claims it is the same. Even if they built the tool in the exact same environment. Bugs may not be reproducible, reports will be harder to parse since end users may not tell you exactly what versions were used for every dependency, etc.
• Even if somehow end users get used to this approach (which not common outside Rust), projects with pre-built binaries need to re-publish anyway, so you get two different approaches to updates.

In summary: we cannot rely on cargo install to be the way to get bug fixes and security patches to end users.

However, in my opinion, the bigger problem is what I mentioned a few replies above: every time you run install, you are trusting an unknown number of maintainers, effectively downloading and running arbitrary code.

[NickeZ]

are people regularly cargo install-ing their upstream binaries over and over again?

It's not exactly about regularly installing, it's more about regularly publishing. Let's say there is a security update in a popular library. There could be thousands of binary projects relying on it directly or indirectly. If Cargo used the original Cargo.lock, all of those binary projects would need to publish an update, otherwise all users will get the old version.

I realize it is awkward and confusing, but switching the default has its own drawbacks. It's not really clear which is best.

Implementing the hint is on my todo list, will hopefully get to it at some point.

Everyone is now teaching cargo install --locked because not doing so breaks the builds. Why would I put anything else in my readmes and dockerfiles? The default leads to broken builds and support tickets.

[est31]

This probably mirrors the experience of others, but cargo-udeps had cargo install broken 3 times within a period of 10 days in late February/early March:

• Feb 29: https://github.com/est31/cargo-udeps/issues/53 caused by https://github.com/TeXitoi/structopt/issues/354
• Mar 5: https://github.com/est31/cargo-udeps/issues/54
• Mar 7: https://github.com/rust-lang/cargo/issues/7974

After the second breakage, I changed the README to suggest usage of the --locked param: https://github.com/est31/cargo-udeps/commit/bcd755d8f7c912270b3ec32cfab37c31a890aac0

Of course, @ehuss 's concerns are valid too, but the state of minor updates potentially breaking everything for tons of users isn't great either.

What about crates.io maintaining a lockfile automatically? It could update the lockfile say daily and if there were any changes, run a basic cargo check on the tool in a docker container to make sure it builds at least and only update the lockfile on success. Of course this (short lived) auto-lockfile could not be part of the package as it'd invalidate all the hashes but instead be offered separately. It could be the default option that's a compromise between safety and tested behaviour. Old lockfiles as well as build logs wouldn't have to be kept/archived for a long period of time, as if you want to pin things, you should still use --locked which would still work as it works now. Only the default would change from using an untested set of dependencies to one which has had basic verifications.

[ojeda]

run a basic cargo check on the tool in a docker container to make sure it builds at least and only update the lockfile on success

Failing to build is not the major issue. Even if you ran tests and they had perfect coverage, the approach would still be non-deterministic and unsafe.

[NickeZ]

run a basic cargo check on the tool in a docker container to make sure it builds at least and only update the lockfile on success

Failing to build is not the major issue. Even if you ran tests and they had perfect coverage, the approach would still be non-deterministic and unsafe.

Exactly, failing the build is the good case, this is equivalent to UB causing a segfault in C applications. It is immediately evident that there is something wrong. The real problem here is when your application is built and used with unknown dependencies, which is more similar to UB causing wrong data being read/written silently. I wish we had deterministic builds by default, but until then I will suggest everyone to write --locked everywhere.

edit:

It's not exactly about regularly installing, it's more about regularly publishing. Let's say there is a security update in a popular library. There could be thousands of binary projects relying on it directly or indirectly. If Cargo used the original Cargo.lock, all of those binary projects would need to publish an update, otherwise all users will get the old version.

I realize it is awkward and confusing, but switching the default has its own drawbacks. It's not really clear which is best.

I think cargo install is trying to solve a problem it shouldn't even try to solve. If I want applications with the latest security updates I will use apt to install them. Not cargo. And I bet all sane distributions uses cargo install --locked because anything else is awkward, confusing, and honestly, not the intent of the original developer and therefore wrong.

[mati865]

If Cargo used the original Cargo.lock, all of those binary projects would need to publish an update, otherwise all users will get the old version.

This won't help anybody who already has the crate installed.

Consider this exaple: Somebody installed crate foo 1.0.0, this pulled dependency bar 1.0.1. On the next day bar 1.0.2 has been released with important security fix. Now people who installed foo on it's release have executable with security issue and other people cannot install foo at all because dependency xyz has broke the build.

That's issue that crate maintainers should take care of by using tools like https://github.com/RustSec/cargo-audit to detect the issue and publish fixed realease afterwards.

[elichai]

Just spent a few hours debugging why I can't reproduce my builds just to find this :(

[euclio]

This behavior broke cargo-spellcheck installation, because hunspell-sys was depending on a yanked patch version of bindgen that didn't have any other compatible patch version.

[mbrubeck]

Another downside of ignoring Cargo.lock by default:

Yesterday, one of my dependencies accidentally released a breaking change without bumping its major version. This broke cargo install for my binary crate. I fixed this by publishing a new version of my crate that works with the new version of the dependency.

But if the upstream dependency maintainers later decide to yank the incorrectly-released version, that will break cargo install for my binary crate a second time (because it now builds only with the yanked version, which is not available when ignoring Cargo.lock), and I'll need to publish yet another emergency fix in order to keep my crate from having broken installs by default.

(Alternatively, I could have updated my Cargo.toml to lock my package to the previous version of the broken dependency, effectively using it as a replacement for Cargo.lock.)

[woodruffw]

Just adding two more pieces of anecdata: this recently broke builds of two Rust CLIs that I depend on. I found it pretty surprising as an end user that checking Cargo.lock into version control is encouraged for "leaf" projects (e.g., CLIs) but isn't actually respected by default by cargo install!

One thing that makes this particularly fraught: Cargo seems to think that rc versions are SemVer compatible with each other, meaning that 1.0.0-rc.2 can be silently selected over 1.0.0-rc.1 unless the maintainer specifically pins the rc with =. As a result, a decent chunk of the Rust i18n ecosystem had a broken dependency tree for over a week.

[RalfJung]

FWIW, making install be --locked by default would have avoided much of the fallout from https://github.com/rust-lang/cargo/issues/9101.

[casey]

I ran into this today. The issue wasn't the build breaking, but cargo install unexpectedly downloading and building newer versions of dependencies when it should have been using cached build artifacts. I was very confused by the default behavior of cargo install.

I think it would be better if cargo install defaulted to using the lockfile, and a new flag was added that enabled the current behavior, something like cargo install --update-dependencies.

[estk]

I wish the change suggested by @casey had been part of the 2021 edition.

[Emoun]

--locked not being the default resulted is cargo-expand and bat builds breaking yesterday:

• sharkdp/bat#1991
• dtolnay/cargo-expand#122

I was astonished (as @NickeZ put it) that bat's lockfile was ignored.

Then again, a semver violation was noticed because of this. However, I would rather prefer users opting into being guinea pigs than forcing it. (e.g. by using --locked by default and --unlocked by choice)

[djc]

I wonder if the solution here could be to both stop publishing Cargo.lock by default (instead, make this opt-in), and then always enable --locked implicitly in the presence of a lockfile.

[Kleidukos]

@ehuss Hi! Is there a final decision on the desired behaviour/where to go from now on? :)

[teohhanhui]

stop publishing Cargo.lock by default (instead, make this opt-in)

That would just be a footgun, i.e. allow breaking by default.

[steveklabnik]

This behavior continues to be a footgun. I keep seeing it pop up, with both new and experienced Rustaceans.

The intent is that cargo install will behave the same whether it is from a registry or a local path.

I can appreciate this idea, but in practice, it ends up feeling different than cargo build and cargo run, which is a consistency I would value higher. It's kinda wild to build a project, it works, so then you try to install it, and it doesn't work. Well, it's actually worse than that: it probably works most of the time because most Rust projects don't violate semver, but then sometimes it will fail whenever a project doesn't. This often happens with git deps, in my experience.

[nixpulvis]

Frankly, as nice as it is to super easily install software from the registry, I wouldn't be offended at all by needing a new sub-command and workflow to accomplish this.

For example:

cargo fetch utc 
cd utc
cargo install

I've chosen to reuse cargo-fetch here (without much previous experience with it) because it feels fitting. But something like cargo checkout could also make sense.

This mirrors the good ol' clone and make workflows we all know and love.

[stevenroose]

Cargo should definitely never secretly download different dependencies than the ones specified by the project. This is definitely a bug. Some projects vet dependencies and others need to pin specific versions because of breakage. Regardless of the reason, if a project specifies the versions of dependencies the project should be built with, Cargo should respect that.

[nixpulvis]

@stevenroose I think the argument is that a project claiming to depend on foo = 1 should be OK with any version of foo in the range >= 1.0.0 through < 2.0.0. However, this is at odds with the concept of a lockfile, which are produced locally to help ensure reliable builds.

One possible solution I could see being a good idea perhaps is to allow maintainers to publish the Cargo.lock file and respect it when published. As the default behavior, I believe that would solve this issue. Maintainers who want to trust upstream dependencies to follow SemVer strictly could opt-in to the current behavior by telling cargo not to publish the lockfile in some way.

Or am I missing something?

[woodruffw]

I commented on this over a year ago, but IMO the problem isn't ignoring the lockfile itself: it's that Cargo treats certain releases as semver-compatible when it shouldn't. In particular, Cargo ignores semver for pre-releases: it treats X.Y.Z.pre.2 as a compatible upgrade for X.Y.Z.pre.1, when prereleases are explicitly allowed to break compatibility.

TL;DR: It's not Cargo's fault that cargo install fails when a package violates semver, but it is Cargo's fault when Cargo itself ignores semver. I think a good fix here would be to change the dependency resolution behavior to treat prereleases as incompatible, potentially add a flag allowing users to opt-into resolution with pre-releases, and retain the current default behavior.

[steveklabnik]

it is Cargo's fault when Cargo itself ignores semver.

Cargo does not ignore semver. Matchers are not currently in the semver spec (though there is a pull request to add them that myself and the other semver maintainers have not taken the time to merge for Reasons), and Cargo implements these the same as everyone else does. Put ^1.0.0-rc.1 into https://semver.npmjs.com/, and see that it also includes rcs. The reason that this is implemented this way is that = for prereleases is not actually useful in the way that people release software and test those pre-releases.

(And I think the pre-release behavior is a red herring; as I mentioned previously, git deps are another area where this can happen quite a bit, and even outside of these more unusual constraints, anyone who releases incompatible versions that are considered compatible by the rules will cause this to happen. Most people are just good about not doing this most of the time.)

[woodruffw]

I recognize that I'm standards lawyering now, but doesn't this section of the spec imply that ^1.0.0-rc.1 ~= 1.0.0-rc.2 is incorrect?

Quote:

A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92, 1.0.0-x-y-z.–.

Unless "might not" here is not intended to be advisory w/r/t dependency resolution. I agree that = isn't ideal for prereleases, but the current behavior is also not ideal.

And I think the pre-release behavior is a red herring

It's definitely a subcase 🙂 -- Cargo can't prevent people from making semver errors, but there are quite a few linked issues above where "matching prereleases" have caused breakage with otherwise well-behaved dependencies. IMO those are cases worth addressing, at least if my reading of the semver spec is correct (which it might not be!)

[steveklabnik]

I recognize that I'm standards lawyering now,

Understood, no problems. This is a common thing that comes up when people talk about semver, you're not alone :)

The key issue is that the spec says "here's what version numbers should mean in relation to each other" and then the matchers have their own ways of matching versions together that don't necessarily mean the same things as what the spec says. They're separate things. Most of the text in the 11 points of the spec were written first, and then various matchers were developed afterwards by implementors. ~ happened, and then ^ was a refinement of that, based on how those 11 points actually played out in practice. And even then, it's not that these things being different is even a problem; you want pre-release versions to possibly break everything, because that's the point of a pre-release, but you also want your pre-releases to be tested, and so you want the ability for someone testing a pre-release to be able to automatically upgrade to the newer pre-releases when possible, to try them out.

All of this works because the lockfile (both in Cargo and in Bundler, and to some extent, npm) allow you to only upgrade when you actively choose to upgrade, which is the normal flow for updating dependencies. That cargo install doesn't respect the lockfile by default means that it has a different flow, which leads to this friction. That's why I believe that it should respect the lockfile, rather than trying to change the semantics of ^.