Tools and beta channel

[nrc]

Essentially nobody is using tools on the beta channel. The effect of this is that tools are hitting the stable channel with fairly serious bugs (e.g., https://github.com/rust-lang/rust/issues/50646, https://github.com/rust-lang-nursery/rustfmt/issues/2695).

We need to do something to address this, either:

• encourage users on beta
• add QA to ensure the tools on beta are bug-free
• reorganise how we map tools versions to Rust channels some how.

[Mark-Simulacrum]

reorganise how we map tools versions to Rust channels some how.

As a potential proposal for this, we could change how we version Rust standard tools, so instead of having Rust 1.26 contain RLS x and rustfmt y, we instead ship RLS and rustfmt on their own schedules (perhaps faster than all of Rust). Essentially, we would gain the ability to promote any given nightly RLS to beta/stable Rust when desired, without shipping a new version of Rust.

This is a fairly major change to rustup, and the related infrastructure, but does seem at least plausibly reasonable: RLS, at least, is in theory not coupled to a particular version of Rust, so to speak.

I don't feel great about this proposal though.

Either way, I think increased QA on beta would be excellent, though I'm not entirely certain what that would look like. I do think this is a wider problem to attempt to solve -- maybe we can sell developers on using beta-channel Rust instead of stable-channel locally, since in theory it has most of the same stability guarantees while greatly enhancing our ability to QA stable releases...

[nrc]

Essentially, we would gain the ability to promote any given nightly RLS to beta/stable Rust when desired

There might be breaking changes to the driver API or save-analysis version that would prevent this working, but those are rare.

[jonhoo]

One question that might help this conversation is: who do we expect to use the beta channel currently? I have yet to find any project that uses the beta channel (beyond listing it in Travis). I suspect few developers (without considering tools) are actively using beta at all.

[nrc]

beyond listing it in Travis

This is the primary use case for the beta channel - we recommend projects use it in their CI so they know 6 weeks in advance if stable is going to break their project (and can file an issue if appropriate).

[llogiq]

I can totally see having a clippy build in travis (though that'd be currently breaky-happy), but how would one test RLS via CI?

[dwijnand]

Perhaps it's useful to enumerate the tools that aren't being sufficiently tested?

For instance I was wondering (because I lack the knowledge): does the use of the beta channel in Travis CI test beta cargo?

The most I know is rust-lang/rust depends on cargo as a submodule in https://github.com/rust-lang/rust/tree/master/src/tools.

[nrc]

They are tested, the problem is there are always some things that are problems, but not caught by tests.