Rust proxies obfuscate changes to the underlying tools when using a stat approach

[dcbaker]

Problem you are trying to solve

I'm a maintainer of the Meson build-system generator, which supports building Rust code in combination with other languages. Meson generally is used to generate ninja, and we write a dependency in the ninja for all targets of a given language to depend on their compiler binaries, so if rustc is updated, ninja will notice and mark all rust targets as out of date on its next invocation and they'll be rebuilt. This works fine when using rustc installed through traditional linux packaging, but rustup doesn't expose these binaries and instead wraps them with proxies, which themselves don't change.

Solution you'd like

As a policy, Meson cannot add always out of date targets that must be run by ninja. The difference in the amount of time a no-op build with ninja takes vs a simple "run rust and collect the version" is non trivial (the former is essentially free). But for other build systems that use make-like file stating we need a file of some kind to stat. I am not myself a rustup user, but I see that rustup can set a per-directory override, so we need to account for that as well, which makes the idea of modifying the proxy when the toolchain changes seem like a non-viable approach.

The most obvious solution to me then would be a simple text file in a well known location that contains the version of the toolchain in use, as well as one that's written to the directory when rustup overrides on a per-directory basis. When the toolchain changes the file is rewritten. I can tell ninja to watch this file as a dependency of Rust targets.

I'm fine with telling my users that if they want to switch to/from local overrides they need to manually regenerate the build rules, or at at least that would be my problem to solve in Meson, and not for rustup to solve

Notes

No response

[rbtcollins]

Your analysis of rustup's behaviour is correct. I don't fully understand ninja's limitations here but I trust you do :).

I would be inclined to suggest querying the version and generating stamp files rather than depending on the compiler binary stat metadata itself. I realise its not as free, but at ~ 2 tenths of a second even on Windows, it should be quite close to the cognitive threshold for users even in a no-op case.

Firstly because the compiler binary itself is not sufficient to determine compiler behaviour - there are libraries involved as well.

Secondly, Rustup's job as a multiplexer means that the rustc version isn't stable even without local overrides: consider the rust channel stable which will change from 1.72 to 1.73 and so on over time.

You could get the rustc file with rustup which rustc and use that to detect when rustup has updated it, but this will give false negatives with (a) local overrides of any sort and (b) changes to rustup default.

[eli-schwartz]

The original question was "yes we're aware that in theory one could write a stamping freshness checker and run it on every build, but this is an unpleasant experience and we'd rather not".

Should we assume the answer is "please do so anyway, there is nothing to change in rustup'?

[rbtcollins]

Ah, went to improve my comment after I noticed @dcbaker had touched on that option and then raced with your new comment. Sorry!

Rustup semantics are such that I don't think we could offer to generate a stamp file for you - we'd have to perform considerable action at a distance and track a list of paths builds have ever occurred in, something that will scale poorly; my edit has included a suggestion that might help your needs under some circumstances, but with quite definite caveats.

One approach you could take is to tie the rustup toolchain name into the build definition in some way, and then set RUSTUP_TOOLCHAIN to that toolchain name: this will override nearly every way that the toolchain can be set, and is a defined and supported interface. That will then let you depend on the file path from rustup which rustc, and thus you get the stat metadata you desire.

[eli-schwartz]

Perhaps it would make more sense to use the first option on that list. Ninja unfortunately still doesn't have a way to set environment variables for a build edge (there is a conceptually related issue here, for env!, resulting in a feature request for rustc to add an --env flag, but I digress).

So setting an environment variable requires running a python script wrapper around rustc for portability, which makes a bit of a dent in that 0.2s and is best avoided if passing a command line option can be done instead.

Ninja will already rebuild targets when the command line changes, so if the toolchain name can be automatically extracted and inserted into the command line as argv passed to rustup-as-rustc, then that may be all we need. Reconfiguring the build could then redetect that and potentially change the command lines in question.

[rbtcollins]

Yes, I agree.

[rbtcollins]

@dcbaker / @eli-schwartz is there anything left for this issue?

[dcbaker]

Can we leave this open for a little bit longer? I would like to write some actual code to handle this before we declare it finished, and I didn’t have time today. I’m planning to get to it tomorrow

[rami3l]

Can we leave this open for a little bit longer? I would like to write some actual code to handle this before we declare it finished, and I didn’t have time today. I’m planning to get to it tomorrow

@dcbaker How's it going with this issue?