Renaming targets might be a breaking change

[imsnif]

Hi all,

I am unsure if this is the right place for this sort of issue and apologize if it isn't. I would also like to stress that I fully understand this issue is not currently actionable, but I want to bring it to the attention of the maintainers in case this is a blind spot. If this has been discussed to death elsewhere (that I could not find) I also apologize for the extra noise.

Recently, through compiler warnings I have been made aware of: https://blog.rust-lang.org/2024/04/09/updates-to-rusts-wasi-targets.html#renaming-wasm32-wasi-to-wasm32-wasip1

Namely, renaming the wasm32-wasi target to wasm32-wasip1. In Zellij, we bundle minor applications that we call plugins - compiled to this target - within our executable. This is how we compile and distribute our software and represents a compromise of various factors. Renaming this target represents a breaking change for us. It means we will no longer be able to compile the same project for the newer rust version, and that if we upgrade our toolchain, we will no longer be able to compile the same project for a (much) older version.

This is not a big deal for us. Making the change should be a trivial search/replace, and we don't absolutely have to support much older toolchains. However, as a user of the language I assumed (perhaps mistakenly) that targets are an external user-facing API. That any non-backwards-compatible changes to them would be considered breaking changes and should only be expected in major version bumps. Since this was not the case here, it gets me a little worried that perhaps either I do not understand the breaking changes policy (even after reading the relevant RFC) or there is some sort of miscommunication going on between the language developers and at least some of its users. Or - of course - this is a blind spot.

In case this is the latter, I wanted to bring it to the attention of the maintainers. Not for this change of course - as I understand it's already very much underway - but at least for future changes.

Thanks for all the work everyone is doing.

[jieyouxu]

This is intentionally a breaking change and why there's a migration timeline https://blog.rust-lang.org/2024/04/09/updates-to-rusts-wasi-targets.html#timeline to help people migrate off the old target name. This included a period of warning if people were still using wasm32-wasi in 1.79-1.81 on stable.

cc @alexcrichton for the wasm targets.

cc @wesleywiser and @davidtwco: do we have any specific target renaming/deprecation/removal policy? FWIW wasm32-wasi/wasm32-waspi1 are Tier 2 targets. The change did go through:

1. https://github.com/rust-lang/compiler-team/issues/607: MCP for initial target rename proposal
2. https://github.com/rust-lang/compiler-team/issues/695: MCP for smoothing transition of the rename

cc tracking issue https://github.com/rust-lang/rust/issues/113364

[imsnif]

For what it's worth, I was going by this when investigating the Rustlang policy: https://rust-lang.github.io/rfcs/1105-api-evolution.html#major-change-renamingmovingremoving-any-public-items

[jieyouxu]

That policy governs stability guarantees w.r.t. language features, not compiler target support. Please refer to Target Tier Policy w.r.t. to compiler target support. In particular, the relevant part is

A tier 2 target may be demoted or removed if it no longer meets these requirements. Any proposal for demotion or removal will be CCed to the target maintainers, and will be communicated widely to the Rust community before being dropped from a stable release. (The amount of time between such communication and the next stable release may depend on the nature and severity of the failed requirement, the timing of its discovery, whether the target has been part of a stable release yet, and whether the demotion or removal can be a planned and scheduled action.)

[imsnif]

Fair enough. Thanks for the link. As a user of the language who is not aware of its internals - I'm not sure I could have easily found this document on my own, or indeed understood that this is what I'm looking for. I looked for "Rustlang breaking changes" and found the above RFC. Maybe I'm the problem - maybe this is clear to everyone else except me.

But from my perspective - without knowing the internals of the tool I'm using - I look at the semver version to understand if I should expect breaking changes or not. And I expect public APIs (including in my view compilation targets, CLI flags, etc.) to not change in a way that would break old code.

This distinction is now clearer to me, thanks for pointing it out. I only wonder what other blind spots I have regarding breaking changes that are not covered in the breaking changes RFC. I am not a Rustlang language developer, simply a user of the language. Do with this feedback as you wish. :)

[jieyouxu]

FWIW I think compiler team did want to clarify/amend the Target Tier policies a bit, since it can be a bit confusing.

[alexcrichton]

@imsnif you might be interested in the target tier policy where the wasm32-wasi target (and wasm32-wasip1) are classified as tier 2 right now.

[imsnif]

Thanks for pointing this out @alexcrichton !

Here was my process and reasoning:

1. I look at Rust's version, see that it's above 1.0.0 and so assume breaking changes only happen in major version changes
2. Seeing this warning, I get confused. I realize I probably view a "breaking change" in a different way than the language developers, and so search for "rust breaking changes", I find this: https://rust-lang.github.io/rfcs/1105-api-evolution.html
3. As a simple user of the language and not a developer of the language itself, I am not aware of distinctions between the Rust standard library, the targets, tiers or any such. My confusion deepens and I open this issue. I am then pointed to the above document, which explains this specific distinction.

Right now, I understand why this change (which I view as breaking) will happen in a non-major version. I am however left with a concern about other potential breaking changes which do not fit into this mold (either the standard library or the target tier system). Things I do not even know to ask about because I don't have the relevant domain knowledge (eg. like here - what's the difference between breaking changes in the standard library and in specific compilation targets based on a tier system).

I look at semver as a way for the language developers to communicate (breaking) changes to me. As the target of this communication and a person who relies on this software heavily, I am currently left quite confused - mostly about the future and the unknown unknowns that I have regarding what constitutes a breaking change and what doesn't.

Do with this feedback as you wish. Thanks again for taking the time to explain this to me and link to the relevant places, and for the work you all do on this great language.

[jieyouxu]

I would like to nominate this for compiler triage meeting to maybe see if we want to amend the target policy docs to better reflect what compiler stability guarantees are made and what is explicitly not covered by compiler stability guarantees, as opposed to library API stability guarantees and language stability guarantees. And also especially about if there's any difference in stability guarantees (or the explicit lack thereof) for different-tiered compiler targets.

@rustbot label +I-compiler-nominated

[workingjubilee]

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes

Unfortunately, SemVer concerns API definitions, it is utterly silent about build requirements.

Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it SHOULD be precise and comprehensive.

From the definition of SemVer, also, it cannot be inferred what is "public API", it must be explicit, and it should be clear.

For RFC 1105, it explicitly addresses the topic of the standard library.

For RFC 1122, it explicitly addresses the topic of the language per se.

I cannot find a place that declares the targets are public API, so while we should probably venture to disambiguate, interpreting SemVer as binding something that is left ambiguous is contrary to the definition of SemVer.

[workingjubilee]

Some questions to answer:

• By being willing to remove targets (ever) or change their definitions in potentially backwards-incompatible ways, we are probably implicitly saying arguments to --target, thus arguments to compiler flags, do not enjoy perfect stability. Where does that start and end?
• For instance, what about -Copt-level=3? Do we guarantee that always works? Do we guarantee that it always has the current effect? What if we add support for -Copt-level=4 and decide -Copt-level=3's current behavior is equivalent to -Copt-level=4? What if we do the opposite, contracting the scale to top out at -Copt-level=2? One could argue that because we explicitly document these arguments to opt-level, they are indeed public API, but is that about recognizing the arguments or the specific effects?
• Meanwhile, the list of arguments to --target is left unspecified for the documentation that describes --target. Is that what exempts the list of targets from such, or is it more a matter of principle here, and we could hypothetically list some targets on the page while noting that they will not be always present and may not have a consistent meaning?
• A philosophical question for the bonus round: What if Microsoft, our primary reference point for the definition of Windows, suddenly vanishes tomorrow? Do we decommission the targets when we determine the "source" is "dead", or do we wait to see if another "reference point" gains consensus? While this happening to Microsoft specifically seems unlikely, every now and then an OS or other important part of a --target does "suddenly vanish" like this, and sometimes has multiple "claims to the throne" in the aftermath, so it is worth contemplating.

[imsnif]

Thanks for the clarifications of how you (and the project?) view semver.

As an outsider who is often unaware of the particularities and (what they view as) internal differences between different (seemingly) public interfaces of the language, but is nevertheless extremely dependent on said language for a non-trivial piece of software - do I have an easy way to determine what constitutes a breaking change for the Rust language? What is the language's position on "will my project compile in the next version?"

Right now I find myself in a position of being extremely conservative about language version upgrades, only opting in to them if they are critical. I unfortunately do not have the personal capacity to parse and understand all of these documents in order to figure out which apply to my use case.

I view semver as a best-effort way for those who do have this domain knowledge to communicate this information to me. Apparently we view things differently. Legit. Is there a different way for me to obtain this information without opening an issue like this one? Or is it just upgrade locally and see?

[hanna-kruppe]

There is a limit to how much nuance can be communicated via a version number. I agree that it would be useful to have an official, discoverable document that users not plugged into the project can read to get a more nuanced understanding of the stability guarantees.

As I am no longer involved in the project I won’t try to answer what should be in there. But let me try to formulate my informed expectations as a user of the language. When I update my Rust version, it almost always just works without any issues, and that’s how it should be. There will be exceptions but it should never be too painful to get back to a working program:

• If cargo build, cargo run, etc. start failing, this should be easy to fix by making small edits to my own code. Renaming a —target or downgrading a #[deny(some_lint)] count as easy, having to patch a library I depend on or needing to rearchitect my code doesn’t. (The type inference change that broke some versions of time a while ago was painful because the fix, while ultimately simple to execute, wasn’t easy to figure out for many people.)
• If the build doesn’t break, then the runtime behavior of the program should be the same as before, modulo bug fixes. Of course Hyrum’s law makes this more aspirational than a hard rule, especially around unsafe code relying on behavior that isn’t documented yet. But I expect that the compiler and standard library won’t be “adversarial” and keep Hyrum’s law in mind.
• I don’t expect any particular details of the generated binaries to remain stable, just that they’re fit for “purpose” (whether as standalone executable or a library others can link to). Regressions in build time and optimization quality are a fact of life, though they should be fixed eventually. There are functional factors outside of the Rust project’s control, e.g., old versions of operating systems eventually reaching their “end of life”. But again, I don’t expect “spurious” or more frequent breakage compared to the rest of the software ecosystem.

These points are intentionally vague with respect to which parts of the project are involved because it’s more like a user story. I also want to stress again that there will always be specific cases where these expectations will be broken: bugs happen, bug fixes can also break things occasionally, in some rare cases there’s no reasonable way to avoid serious breakage, and occasionally a user unwittingly managed to depend on details that can’t be considered (or post-hoc promoted to) a “public interface”. But the stated goal of Rust’s stability guarantees is to keep updates painless so everyone can reasonably be expected to update at a steady pace. For this to work, updates have to be actually painless for the vast majority of users in the vast majority of cases.

[workingjubilee]

I don't speak for everyone when I note that SemVer is inadequate for the purposes that people hope it will serve, I merely am quoting from https://semver.org/ because it is that spec that defines SemVer as an inadequate tool for many tasks.

In any case, it is the full intention of the WASI spec to change its definition, because it is prototype software, in effect. Repeatedly. These redesigns are no minor difference. Each redesign can in fact break running actual software using these targets. Even minor redesigns which weren't considered "preview version bumps" were breaking software. And for the preview version differences, each one may involve rewritten system APIs, subtly different build requirements, or even new binary formats. Effectively everything was on the table for breakage already for the wasm32-wasi target, except for the ISA.

That is why wasm32-wasi is being iced for now, as it would be much more strange to claim we are respecting any kind of stability or compatibility... forward or back... while tolerating a target being three or more wildly different versions over the span of a couple years. The intention is to reinstate the unqualified wasm32-wasi tuple when WASI reaches 1.0, instead of being, itself, on 0.1... well, 0.2 for the wasm32-wasip2 target. That may happen to p3, or it may be p9. No one has committed to anything on that front, that I am aware.

If you, @imsnif, would be quite happy with the target's name remaining the same, but potentially rapidly acquiring radically different meanings, causing your builds to break in a boutique way, instead of being something easily fixed like renaming the argument to --target? Then I suppose I have lost a bet.

You can in fact curse my name for being the one that said that programmers... like you... are going to expect more stability than the WASI target wanted to offer. And I did indeed say that such meant that we shouldn't accept a target having a "whatever" definition that is suddenly going to be redefined later in effectively unpredictable ways. My reasoning was that this way you can at least see the target definitions changing, instead of it suddenly ambushing you someday when WASI finally canonizes some more-stable definition and the target is rewritten overnight.

[imsnif]

Hey @workingjubilee - so, just to get two very important things out of the way:

1. I am not here to curse or even blame anyone. I realize you may be hyperbolic, but even if only for the gallery here - I want to stress this point. My intentions here were primarily to understand the Rust project's approach to breaking changes from my perspective as a user and to a much lesser extent to raise the possibility that my use-case might be a blind spot. I appreciate the hard work and considerable thought that I am sure is being put into this project.
2. I very much agree that SemVer is inadequate for the purposes people wish it would serve. I would even go a step further and say that I feel it is inadequate for almost anything. Namely because the definitions of what constitutes a (breaking) change, an API, a dependency, a dependent, a user, a developer/maintainer, a piece of documentation and responsibility are extremely subjective. As we see partially demonstrated in this thread. However, it's the system we've got - and I personally feel it's a good one to communicate intent.

I appreciate your detailed explanation of the reasoning behind the target name change. I understand you did not wish to convey even the appearance of stability for that which is clearly a moving target that can change in unexpected and subtle ways. I cannot speak for other programmers, only for myself when I say I would definitely have preferred the target name remaining as it is.

As an early adopter of WASI, I am aware of this volatility and have chosen to accept it. From my perspective, any such behavioral change is extremely subtle. To the best of my knowledge, we have not encountered any such noticeable change over the lifetime of the application.

The renaming of a target on the other hand is something that has a significant effect on us. It breaks both our CI and custom tooling (eg. we have loaders that compile plugins and rely on the result being in a certain folder - this change would represent a runtime error for them). To go further, our CI has some code-paths that run only when publishing our package to crates.io. If for some reason the toolchain is not picked up during this process, the target name change could cause us to fail to publish a version while everything else passes. Leaving us in the unenviable anxiety-inducing place of having a partially published version with dozens if not hundreds of complaints pouring in in real time of things not working*.

So in short - while my use-case may not be representative here, and while maybe this preference would be wildly different for other programmers - I would definitely prefer what I interpret as public facing APIs not to ever change outside of a major version. Sorry for making you lose the bet. :)

and just to be clear on this one - a lot of this has to do with problematic tooling/practices and things that could definitely* be improved on our side, I'm not avoiding responsibility here or trying to blame anyone else for it

[hanna-kruppe]

As an early adopter of WASI, I am aware of this volatility and have chosen to accept it. From my perspective, any such behavioral change is extremely subtle. To the best of my knowledge, we have not encountered any such noticeable change over the lifetime of the application.

You haven't seen any notable changes because the wasm32-wasi Rust target has been frozen as WASI 0.1 / preview 1 the entire time. Changing that target in-place to mean the target now known as wasm32-wasip2 would have been pretty disruptive:

1. The Rust standard library and various third-party libraries have to be updated to switch to the wasip2 interfaces. I don't know how far along this is today, but the blog post you linked in the issue said the standard library support was very incomplete at the time the target was introduced.
2. The wasm runtime / embedder has to provide the new interfaces instead of the old ones. Wasmtime has been pretty quick with this, but programs that use wasmtime as library still have to use that capability, and outside of Wasmtime support is still pretty sparse (e.g., node.js has some support for WASI 0.1 but not for 0.2. and likewise for several JS shims that support running WASI modules in the browser).

Indeed a quick code search through the Zellij repository suggests you would have encountered both problems: you're embedding via Wasmtime but only creating a preview1 environment (no results for preview2 anywhere), and if the wasi crate occurring in your Cargo.lock are actually used for some plugins then these are versions predating support for preview2.

So redefining wasm32-wasi would have been more breaking for more people. It would have been better to name the preview1 target wasm32-wasip1 from the start, but for lack of a time machine the only option that wouldn't break anyone's use case would be to permanently keep the wasm32-wasi target as alias for wasm32-wasip1 and find a new name for a future "WASI 1.0" target. But this is not a great option either, because it would mean (1) never removing the wasip1 target, even if it becomes completely irrelevant in the future, and (2) forever having a very prominent, misleading target name that every user has to be steered away from. This is in fact what's done for stable, deprecated APIs in the standard library. However, the trade-offs for doing this with library APIs vs. targets are different, so the policies are different.

[hanna-kruppe]

The renaming of a target on the other hand is something that has a significant effect on us. It breaks both our CI and custom tooling

By the way, I feel your pain about this. I also have a project where, as part of building a Rust program natively, I have to compile some other Rust code to wasm and find the resulting module. As you allude to, there are more and less robust ways to go about this sort of thing. I've spent a lot of time thinking about the best way to do it in my case, but this machinery is still at the top of the list of how a Rust update could break this project. Indeed it already happened once, and at the time I was very thankful to my past self for recognizing a certain assumption was being made and writing the code in a way that failed in an obvious way when this assumption was broken. So my advice would be:

• If you have to script cargo builds and interact with the build outputs programmatically, expect somewhat less stability than the Rust code that's being compiled can expect, and program defensively to proactively catch changes that would break your project.
• But also do try to use interfaces that are explicitly intended for external programs to consume, e.g., get paths of build artifacts from Cargo's JSON message format instead of manually constructing a path into the build cache.
• Don't delegate this stuff to a third party tool/library if you can help it. Something will break eventually and you'll be happier if you can fix it yourself.

To be clear, the above is my personal position, I don't claim that it reflects any existing policies (formal or informal) within the Rust project. It just seems prudent based on facts I know, e.g., many aspects of the build cache contents are explicitly called out as "internal and subject to change", rustup doesn't necessarily ship every target and every toolchain component forever, the target tier policy allows targets to change as needed and get removed if they become unmaintained, etc.

[imsnif]

Indeed a quick code search through the Zellij repository suggests you would have encountered both problems

Without getting too deeply into our architecture, SDKs, usage of wasi and the way we distribute plugins, this would have had considerably less impact on us, and any impact it would have had would have been more easily mitigated on the application level.

So redefining wasm32-wasi would have been more breaking for more people. It would have been better to name the preview1 target wasm32-wasip1 from the start, but for lack of a time machine the only option that wouldn't break anyone's use case would be to permanently keep the wasm32-wasi target as alias for wasm32-wasip1 and find a new name for a future "WASI 1.0" target.

Or aliasing it to wasm32-wasip1 with a deprecation warning and retiring it in the next major version. Again though - I'm not here to bike-shed the Rust compiler. Probably everyone participating in this thread has more relevant domain knowledge than I do and I trust the judgement of those making these decisions regarding the trade-offs.

* If you have to script cargo builds and interact with the build outputs programmatically, expect somewhat less stability than the Rust code that's being compiled can expect, and program defensively to proactively catch changes that would break your project.

* But also do try to use interfaces that are explicitly intended for external programs to consume, e.g., get paths of build artifacts from Cargo's [JSON message format](https://doc.rust-lang.org/cargo/reference/external-tools.html#json-messages) instead of manually constructing a path into the build cache.

* Don't delegate this stuff to a third party tool/library if you can help it. Something will break eventually and you'll be happier if you can fix it yourself.

Thanks for the recommendations. The relevant example I gave above cannot benefit from these unfortunately because we need to run the actual cargo commands in order to show their output as-is to the user in a dedicated terminal window as part of the development environment. Using the programmatic interface would mean we would be in charge of rendering on our own which I would very much prefer to avoid. But really - this is just one example. Just like I'm sure a lot of thought was put into the original issue in this thread, so has a lot of thought been put into the issues I mention on our application side.

I do agree about defensive programming - my solution is a combination of this thread (to understand how the Rustlang maintainers see breaking changes) and being very conservative about Rustlang version upgrades.

Thanks again for your advice.

[hanna-kruppe]

Or aliasing it to wasm32-wasip1 with a deprecation warning and retiring it in the next major version.

There is not going to be a next major version of Rust. Between editions enabling backwards-incompatible changes to the language without actually breaking anyone's build or splitting the ecosystem, and many minor technically-breaking changes not being considered semver-major changes (otherwise we'd be at Rust version 82.0.0), there isn't any reason to accept the enormous pain a "Rust 2.0" would bring (think: Python 3). So the choice is basically between keeping something deprecated or non-functional as long as Rust exists, or removing it at some point in the 1.x series (possibly after a very long deprecation period).

[davidtwco]

In addition to the wording quoted earlier by @jieyouxu, we also say the following about the availability of a target in the target tier policy:

The availability or tier of a target in stable Rust is not a hard stability guarantee about the future availability or tier of that target. Higher-level target tiers are an increasing commitment to the support of a target, and we will take that commitment and potential disruptions into account when evaluating the potential demotion or removal of a target that has been part of a stable release. The promotion or demotion of a target will not generally affect existing stable releases, only current development and future releases.

Unfortunately, it's just not really possible for us to avoid making breaking changes to targets - sometimes targets just don't work anymore (e.g. asmjs targets), aren't supported by their vendor anymore (e.g. Windows 7), or cases like this where the platform isn't stable and changes. We try our best to can do put migration timelines in place and try to make people aware of these changes, as in https://blog.rust-lang.org/2024/04/09/updates-to-rusts-wasi-targets.html#renaming-wasm32-wasi-to-wasm32-wasip1, but that's about the best we can do given that the stability of a targets is often external to the Rust project.

[apiraino]

Linking the discussion happened on Zulip during T-compiler triage meeting

@rustbot label -I-compiler-nominated