suggest new bootstrap technique

[aidancully]

Bootstrapping the rust compiler requires a working rust compiler, which is currently obtained via a snapshot directory. I'd like to suggest, instead, that any object files compiled from rust-language source be distributed as LLVM IR code, and the LLVM infrastructure be used to compile those to native objects as part of the stage0 build. If this sort of approach is possible, it would remove the dependency on rust for a working rust compiler to build the rust compiler, simplifying deployment to new platforms.

[sfackler]

Isn't IR platform dependent?

[aidancully]

It's outside my wheelhouse, but according to this stackoverflow answer, IR can be written in a platform-independent way. So long as the places the rust-language source interacts with the platform are kept behind a C-compatible abstraction boundary, I think this technique can be made to work.

[Thiez]

Wouldn't it be easier to just crosscompile rather than complicating rustc codegen to generate target-independent IR?

[aidancully]

Cross-compilation is not that easy. I didn't have a spare box to put a Linux distribution on, and if I did, there would have been a learning curve in getting the cross-compiler working for my target. On the other hand, rustc codegen should already generate mostly target-independent IR, right?

[Thiez]

Maybe, I wouldn't know. I think generally it would not be unreasonable to require people who wish to port rustc to a new platform to have access to at least one of the currently supported targets, especially since it is so easy to run linux on a VM. And when we reach Rust 1.0, perhaps it would be possible to make sure the compiler uses only stable features, and then porting the compiler to a platform once should ensure it can compile any future version of the compiler.

[Gankra]

@Thiez I'm not sure how stable features applies here. The compiler could easily use new stable features that old compilers don't know about. However there should always exist (even today) a chain of compilers that you can compile to eventually migrate your build to master. It just might be hundreds of compilers long.

[brson]

Regardless of whether IR can be written in a platform-independent way, that IR necessarily calls into platform-specific functions, so to avoid the bootstrapping problem you would have to have some runtime abstraction layer for everything that rustc depends on written in not-rust.

Bootstrapping new platforms is not a common activity so I'm not too enticed to make heroic changes to the bootstrap to accommodate.

I've heard in the distant past concern that some freedom-loving platforms might not like bootstrapping from bins, so bootstrapping from IR might please them, but even Debian has a process for this so I'm not sure there's a valid use case there.

[thestinger]

I've heard in the distant past concern that some freedom-loving platforms might not like bootstrapping from bins, so bootstrapping from IR might please them, but even Debian has a process for this so I'm not sure there's a valid use case there.

Debian isn't going to be any happier if it bootstraps from IR. It is not different than machine code / assembly from that perspective.

[andrewchambers]

Even GCC does not bother with this. Its reasonable to expect a working cross compiler running on something like linux.

@aidancully Sorry to sound harsh, but if your target had real value there would still be a ROI for purchasing or renting a linux machine.

[aidancully]

@andrewchambers, I read the situation differently than you do.

a) Rust isn't anywhere near gcc's level of popularity, platform vendors like Intel and ARM themselves devote significant resources to ensuring that gcc runs well on their platforms, so that a binary would always be available. Rust is not (yet?) in that position, and must rely on volunteer effort to port to alternative platforms. And since the language and compiler are evolving so rapidly, the compiler needs to be regularly re-ported to second-class platforms. I can't say for sure today, but before GCC achieved ubiquity, they absolutely worried about bootstrapping (for example making sure they were bootstrappable by things like Sun's compiler suite). Granted, the world has changed a lot since then, VMs are much easier to set up, and cross-compilation is easier than it used to be. But if the build process can be designed such that a VM isn't necessary (and if it isn't too onerous to do so), then why a VM be forced onto porters?

b) To demonstrate real value, there at least needs to be a proof-of-concept. Right now, the proof-of-concept is harder to demonstrate than it might need to be.

For what it's worth, I filed this partly to gauge interest in the idea. I'm willing to investigate this solution, but only if it seems like there's a possibility of a payoff...

[andrewchambers]

I can't imagine a realistic situation where there is no access to a machine capable of cross compilation. Somehow you believe setting up a VM or acquiring a working host is too onerous. I would suggest that porting rust and llvm is already more onerous than than the 5 minutes it takes to provision a linux machine to get a bootstrap compiler.

Before software capable of compiling GCC was free and common, they needed to worry to ensure other people can get access to a gnu system. The software for compiling rust is already free, common and open source, so why worry?

[thestinger]

LLVM IR is not at all platform independent. There isn't an actionable issue here so I don't see a reason to leave this open.

[thestinger]

PNaCl provides a portable bytecode and rustc is able to target it... but it's not going to run without someone working to expose everything it needs in the execution environment. At that point you'll have written a portable OS API usable for running lots of existing software.

If someone really wants to do that then nothing is stopping them - but it doesn't belong in this bug tracker. It's not at all in the scope of the project and there's no advantage to doing it in this repository. It doesn't seem very Rust-specific either.

[aidancully]

Thanks for the feedback.

[andrewchambers]

@thestinger Well, to be fair, a compiler needs very few syscalls - reading and writing files, and allocating memory.

[pinobatch]

@andrewchambers GCC can in theory be bootstrapped as follows:

1. Use whatever C compiler you have to build GCC 4.7, the last version before the 2012-08 conversion to C++
2. Use that to bootstrap more recent GCC

I can't imagine a realistic situation where there is no access to a machine capable of cross compilation.

The situation would be a protocol to avoid self-propagating compiler trojans like those described in "Reflections on Trusting Trust" by Ken Thompson. When there exist multiple independent implementations of a particular language, the diverse double-compiling (DDC) defense described by David A. Wheeler can be used. But because there are no other implementations of the Rust language, the process must start with a DDC-verified compiler. Thus there needs to be some way to go from source code in a diversely implemented language, such as C++, to something that can compile Rust.

[Thiez]

The rust compiler used to be implemented in Ocaml. You are free to bootstrap all the way from that version to master if Ken scares you, unless you believe the Ocaml tools have code hidden inside them to detect whether they are compiling a Rust compiler, but at that point you're probably excessively paranoid.

Regardless, there is a project to implement a Rust compiler in c++, so soon (tm) you will be able to bootstrap that way as well.