search

rust-lang / rustup

The Rust toolchain installer
https://rust-lang.github.io/rustup/
Apache License 2.0
6.15k stars 883 forks source link

Support `stable-X` as a version descriptor #2291

Closed thejpster closed 1 year ago

thejpster commented 4 years ago

Describe the problem you are trying to solve

In Rust-Embedded land, we are struggling with the management of our Minimum Supported Rust Version. That is, the promise we make to our users that compilers all the way back to our MSRV will compile this code, and we won't use any features stabilised after that compiler was released.

Currently we set an MSRV like a stake in the sand, and then time moves on and the stake gets further and further away. One of our crates has an MSRV of 1.30 - thats 72 weeks old at the time of writing. What I'd like to do is bring that MSRV along with us - trailing it behind us on a piece of string.

Describe the solution you'd like

Basically in my CI files I want to "rustup install stable-2", and if stable is 1.40, it would install 1.38. If stable was 1.51 it would install 1.49. Then I can set my Travis CI matrix to build on:

And I never have to touch the CI file again.

Notes

At the moment, to achieve a rolling MSRV we would have to touch the CI file every six weeks when there's a new stable release. That or rustup install stable, grok the current version with some grep/sed foo, subtract two minor versions (not easy in Bash) and then install that version.

kinnison commented 4 years ago

It's an interesting idea, certainly. However it wouldn't fit directly with rustup's current model. That's not to say it couldn't be made to work, just that it'd be awkward.

If you're insistent on doing it via a Travis matrix then it'd have to be rustup or else travis would have to have some special behaviour to achieve that. Currently rustup is able to just go and ask static.rust-lang.org for channel data. I wonder if, perhaps, it might be sensible to have a rolling set of stable-{1,2,3,4,5} channel symlinks on there?

kinnison commented 4 years ago

At that point, we could add support for the requisite channel name shape to rustup and wouldn't need special handling otherwise.

thejpster commented 4 years ago

I guess the Debian equivalent would be old-stable and old-old-stable, if the numeric component causes an issue.

kinnison commented 4 years ago

I think oldstable and oldoldstable are more plausible just because - is used to separate the elements of toolchain names. In theory ~ could be used too but that introduces a whole host of usability issues potentially. @pietroalbini If I were to propose adding some channel symlinks (or however we deal with stable/nightly vs. the numbered/dated equivalents currently) which approach do you think would fly?

rbtcollins commented 4 years ago

Do we know from channel data the stable version? We could do the arithmetic easily enough in rustup to synthesise new pseudo channels, no?

kinnison commented 4 years ago

Such synthesis would change the download-channel-install-channel model which IMO would be moderately awkward; though as I said before yes it'd be possible. However the ideal would be if the pseudochannels were handled on static.rlo, otherwise we have to start faffing with working out which patch level is the newest for a version (e.g. if stable is 1.35.0 and the user wants stable~1 do we install 1.34.0 or the more useful 1.34.2 ?) So many roundtrips would make rustup update expensive and slow. And yes, I know we do that for nightly backtracking but that's only one dimension (date) rather than more (channel / minor / patch)

pietroalbini commented 4 years ago

I think a good approach would be to create new manifests for 1.MINOR.x (like 1.43.x, 1.34.x...) that always point to the latest patch release of that minor version. This would allow to:

tesuji commented 4 years ago

Yeah, I like using stable~<number> too. Feels like using git reset HEAD~<number>.

tesuji commented 4 years ago

if stable is 1.35.0 and the user wants stable~1

The good default to me is using latest patch release. As patch releases often backport soundness and compatible/build fixes.

Edit: minor -> patch

thejpster commented 4 years ago

if stable is 1.35.0 and the user wants stable~1

The good default to me is using latest minor release. As minor releases often backport soundness and compatible/build fixes.

I'd call that the patch release, as the 35 in 1.35.0 is the minor, and the 1 is the major. But yes, I can't see why anyone would want to not pick up the highest patch release for a given major/minor pair.

kinnison commented 4 years ago

So I like both @pietroalbini 's idea of MAJOR.MINOR.x being the latest PATCH for MAJOR.MINOR though I'd note that might confuse people who do numeric ordering and assume the dots separate numbers. and I like the idea of stable~1 being "one back from current stable". The .x links have the advantage that they're only updated on a release of that MAJOR.MINOR series, but the stable~n would have to be updated every stable release. If we do the latter, I don't think we should do it for beta or nightly though.

pietroalbini commented 4 years ago

I'd prefer not to have stable~N as actual manifests on static.rlo (keeping them up to date is going to be a bit of a pain). Could we go with MAJOR.MINOR.x on static.rlo, and implement stable~N on the rustup client?

kinnison commented 4 years ago

I guess we could have a go. I don't like the double-downloading of manifests but I suppose if the user is opting in then they're prepared to pay the cost. So we define stable~N to mean major(STABLE).(minor(STABLE)-N).x and then fetch that? Will we backfill static.rlo with the MAJOR.MINOR.x for all the releases? Presumably that'd be fairly cheap.

pietroalbini commented 4 years ago

Yeah pretty sure we can backfill static.rlo.

kinnison commented 3 years ago

We now have major.minor channel files present on the dist server, so we're a step closer to being able to do this.

If someone wanted to have a go at implementing the channel parsing and handling the double-manifest-download then we can see how it works.

rami3l commented 1 year ago

@rbtcollins This seems a bit complex, and I don't have a concrete timeline, but I still want to give this one a try. Could you please mark this issue as claimed?

rami3l commented 1 year ago

Before actually trying to implement this change, I'll try to share my thoughts on this issue below:

De-sugaring

It seems that the main use cases for this stable-DELTA syntax is in CIs, so at which level should we de-sugar it?

IIRC, in today's model, stable, 1.71 and 1.71.0 are 3 separate toolchains (assuming the current stable is on 1.71.0):

  1. stable will become 1.72.0 weeks later.
  2. 1.71 will stay on 1.71 but still upgradable (e.g. from 1.71.0 to 1.71.1).
  3. 1.71.0 will get pinned to 1.71.0 forever.

To me, it will be too weird to actually make stable-DELTA an upgradable toolchain just like stable, so I'm considering to define the semantics of stable-2 to be exactly the same as 1.69. That is, I don't expect anyone to actually perform a +0.1 upgrade on those.

Does this sound acceptable? (cc @kinnison @joshtriplett)

Abandoned alternative plan ~~As currently envisioned, `stable-2` will be a separate toolchain installation that:~~ ~~- Are distinguished from `major.minor` toolchains.~~ ~~- Will receive `+0.1` updates just like `stable`, but always maintain a `-0.2` difference from `stable`'s actual version.~~ ~~- Can get out of date (if installed before a `stable` version bump), and thus sometimes require `rustup update`.~~ ~~And since~~ ~~> I'd prefer not to have `stable~N` as actual manifests on static.rlo (keeping them up to date is going to be a bit of a pain).~~ ~~This means that we have to locally maintain "virtual channels" (a.k.a. "pseudo channels") based on `actual(stable) - 0.2`'s manifest.~~

Full syntax

The current syntax for a toolchain spec is:

CHANNEL[-YYYY-MM-DD][-TARGET]

... where:

... and I suggest to add a possibility for the 2nd part: an unsigned integer of at most 3 digits, so that it becomes:

CHANNEL[-YYYY-MM-DD|-WWW][-TARGET]

This would then accept the following (still assuming the current stable is on 1.71.0):

[^sem]: Allowed by the syntax, but might cause semantic errors.

thejpster commented 1 year ago

How I would direct rustup/cargo to do a build with the resolved toolchain? Will cargo +stable-2 build work?

rami3l commented 1 year ago

How I would direct rustup/cargo to do a build with the resolved toolchain? Will cargo +stable-2 build work?

@thejpster It is still a bit too early to talk about this. As I see it, there are two ways of defining the semantics of stable-2:

  1. Making it a shorthand of 1.69 (if stable is on 1.71), which won't receive any +0.1 updates. If this happens, then with stable-2 you have just installed 1.69 and nothing more, since stable-2 is not actually a channel in this case. The actual de-sugaring happens every time the syntax is used, so every time you are using something like cargo +stable-2 build, you are sure that you are actually building with the current stable-2 (but that also means it will instead require 1.70 if the stable version has bumped).

  2. Making it a "virtual channel" installed as separate toolchain. If this happens, then stable-2 behaves just like stable (but 2 versions behind the current stable): they are distinguished from major.minor toolchains, will receive +0.1 updates, can get out of date (and if that happens you need to update them with rustup update), and finally, cargo +stable-2 build requires a stable-2 toolchain installation to work.

I'm still hesitant about this, initially I wanted to go with 1. (as posted above), but after some discussions with other devs it seems to me that 2. is a better option. After all, these two semantics should be equivalent on CIs, but what if someone wants to do this on their own dev machine?

But on the other hand, I don't currently know if this virtual channel thing does exist. If it doesn't, I might have to invent the wheel myself :o

There are other similar things to decide, for example:

[^1]: I was suddenly reminded of my time spent on SICP... What about -2+stable? 😂

djc commented 1 year ago

I think I agree that (2) would provide a better UX, although it would be more work to implement.

(I also agree the minus vs connector thing needs to be thought about.)

thejpster commented 1 year ago

Thank you!

IIRC Debian use oldstable and oldoldstable.

rami3l commented 1 year ago

Thank you!

IIRC Debian use oldstable and oldoldstable.

@thejpster Thanks for the help!

This is indeed very clear without using any weird characters! I'll consider using this name or at least some variant of it :)

djc commented 1 year ago

I think there will also be use cases for old-old-old-old or even old-old-old-old-old-old-old-old, so I'd probably try to get a number in there rather than relying on repetition like this.

rbtcollins commented 1 year ago

There are two design points referenced in the recent message.

1) what name should rustup accept to specify one of these stable + offset things, 2) are they real channels or virtual.

1) I think we should accept stable-${decimal} : debians versioning schemes are very rigorous but also influenced and limited by compatibility with early versions of dpkg and apt, and we are not constrained by them.

2) per https://github.com/rust-lang/rustup/issues/2291#issuecomment-738317030 there are now manifest files for major.minor releases which get updated. A bit of history: we used to have channels like stable and exact versions but not major.minor. This meant you couldn't install rust 1.72 via metadata - rustup had to decide you meant 1.72.0 and get you that instead. This means that we already have actual real channels for each concrete floating version that might be referenced in an MSRV if we can name them.

The proposal linked above is that when rustup encounters a name with a version offset (@kinnison used ~, I think we should use - as ~ has different semantics in Debian and it would be confusing to use that), rustup will examine the stable version number, subtract the version offset from the minor part, discard the point part, and then use that channel as the toolchain.

The question about whether we end up with e.g. a toolchain on disk called stable-2 is a good one. I propose we don't do that and take the option (1) described in https://github.com/rust-lang/rustup/issues/2291#issuecomment-1690899882

What do you think of this possible bit of documentation for the feature.

# MSRV 'last-N releases support'

Rustup can support MSRV for projects that want to build up to N releases back from the current stable release by building using the syntax `stable-N`. For instance `rustup toolchain install stable-3`. This will:
- install the stable toolchain if not installed (this provides the anchor point by which `stable-3` is calculated
- install a toolchain `MAJOR.MINOR` with the version number determined by subtracting 3 from the stable version numbers minor part and dropping the point part

When building for MSRV support, use a [toolchain override](...url skipped) of `stable-N` to build against the current major.minor toolchain as described above. If your stable toolchain is out of date, this may build with an older major.minor than you might expect. Point releases on the major.minor toolchain are only applied when a `rustup toolchain update` is performed - the same as for any other toolchain.

Reasons why I propose this:

rami3l commented 1 year ago
  1. I think we should accept stable-${decimal} : debians versioning schemes are very rigorous but also influenced and limited by compatibility with early versions of dpkg and apt, and we are not constrained by them.

I'm still wondering if there will be a 2.0 moment of Rust, and how we might design this feature to be future-proof in that respect.

  • reduces the number of distinct toolchains people need: if they have 1.69, and 1.69 is needed for a stable-N scenario, their 1.69 is reused. We have open bugs about preventing additional downloads of toolchains, so it makes sense to me to take that into consideration in new feature development.

One downside of this approach, I imagine, would be too many old toolchains on one's disk if they choose to install locally. 1.69, and 1.70 after that... There doesn't seem to be a "garbage collection" mechanism.

weihanglo commented 1 year ago

I'm still wondering if there will be a 2.0 moment of Rust...

When there is Rust 2.0, I believe rustup is very likely to ship breaking changes along with that. Thus not an issue from my pov.

djc commented 1 year ago

I wonder if we can support this without mandating that the stable toolchain is installed? Or do we think that ~everyone has stable installed anyway so there's not much point?

rami3l commented 1 year ago

@djc I think @rbtcollins wants stable to be installed as an anchor point so that we can effectively know what stable-2 actually means (so that stable-2 is always 2 versions behind the installed stable rather than the real stable).

This IMO is also different from my original proposal, in which I wished to resolve stable on the fly and calculate the actual version based on that, but I think it is somewhat more consistent.

Ideally we can do both based on whether stable is installed, but I'm afraid that will cause more confusion than necessary...

rbtcollins commented 1 year ago

I realise I didn't cover one of the design constraints. Because rustup proxies are in the critical path for build time, we need a design that doesn't pay (much) overhead, and that works when the internet is down.

The (much) overhead - its probably ok to run rustc, or parse the stable manifest or both, when a proxy is invoked with no RUSTUP_TOOLCHAIN set. Once set, it needs to be set to a concrete value that won't trigger resolution again.

I'm not sure what I'm proposing is necessarily the best, but it seems ok at current thinking to me.

Using stable-2 as a virtual-channel would have similar properties with the following differences:

epage commented 1 year ago

Something I've adopted in my projects and am also starting to use in cargo is to have RenovateBot automatically update my MSRV.

e.g. see https://github.com/rust-lang/cargo/pull/12381

Would something like that existing lower the priority of implementing a feature like this directly in rustup?

djc commented 1 year ago

An alternative possible design is that rustup should expose +msrv which could read the rust-version from the project's Cargo.toml and run that. Maybe simpler and more directly solves the use case? I suppose this would be roughly analogous to using the rust-toolchain file.

rami3l commented 1 year ago

An alternative possible design is that rustup should expose +msrv which could read the rust-version from the project's Cargo.toml and run that. Maybe simpler and more directly solves the use case? I suppose this would be roughly analogous to using the rust-toolchain file.

@djc Yes, that looks interesting, which could even free us from caring about stable.

djc commented 1 year ago

The tricky part is how this works in workspaces, which don't natively have a rust-version. They could have a workspace.package.rust-version, which could alleviate the problem, but the interactions with --manifest-path might be surprising? I guess cargo +rust-version --manifest-path=foo/Cargo.toml could not pick up the rust-version from foo/Cargo.toml -- but then we already have the same problem today with rust-toolchain files?

thejpster commented 1 year ago

Forgive me if I quote myself:

Currently we set an MSRV like a stake in the sand, and then time moves on and the stake gets further and further away. One of our crates has an MSRV of 1.30 - thats 72 weeks old at the time of writing. What I'd like to do is bring that MSRV along with us - trailing it behind us on a piece of string.

djc commented 1 year ago

@thejpster I'm not sure exactly what you are trying to convey. Is it that you want the +stable-X to be trailing along without you having to update the rust-version? If so, don't you want your Cargo manifest metadata to be correct as the MSRV increases over time?

thejpster commented 1 year ago

My OP I think dates before rust-version existed so yes, we wanted MSRV to move without changing anything in the repo.

djc commented 1 year ago

So how/when are you going to the rust-version in your crate's Cargo.toml? Or are you going to publish your crate without rust-version even though there is some limited MSRV support?

thejpster commented 1 year ago

I don't know that anyone has thought about it. I would probably publish without a rust-version and say in the README "If you're more than three versions behind stable, this might not work"

epage commented 1 year ago

If the answer is "don't set a rust-version", then this feels too specialized of a feature.

The ideal we should be working towards is setting rust-version

thejpster commented 1 year ago

Then I'm happy to declare this 3 year old ticket as OBE and close it.

djc commented 1 year ago

IIRC @epage has used Renovate to send automated PRs to advance the rust-version? Might be a good option.

rbtcollins commented 1 year ago

Yah - rustup toolchain files are not a good fit for MSRV management; that belongs with cargo.