Multiple Codegen Backends for same target

[DrewRidley]

Problem

Hey all, when I attempted the following toml configuration:

[profile.dev.package."*"]
codegen-backend = "llvm"

[profile.dev]
codegen-backend = "cranelift"

I noticed that all of the crates were being compiled with cranelift and not LLVM, even the currently active crate. The expected behavior here is that the current active crate is compiled with cranelift, but all dependencies LLVM.

I think this has a lot of use and merit because cranelift is being developed and targeted as a backend specifically for quickly prototyping projects. For gamedev, it is not suitable to use cranelift to compile the entire game engine and other dependencies due to a lack of optimizations. Furthermore, dependencies only need to be compiled once, and incremental builds rarely revisit these dependencies.

Proposed Solution

The proposed solution is for cargo to allow usage of multiple backends for different targets in a single toml file, and to correctly respect the desired backend. The independent codegen units could be linked as usual after compilation, and there shouldn't be any UB as a result of using multiple backends. This would probably still have to be unstable because of the lack of guarantees regarding calling conventions used from different backends. Despite these risks, I think it is still worth giving the developers the choice to do this.

Due to potentially conflicting usage of lto = fat, cargo would have to emit a warning or error in cases where there are multiple different codegen backends for different targets. cross-crate LTO would not be possible in this case.

Notes

No response

[weihanglo]

Thanks for the report!

I noticed that all of the crates were being compiled with cranelift and not LLVM,

How did you verify this? From the verbose output of cargo build --verbose, did you see rustc invocation for the primary package contains an option of -Z codegen-backend=cranelift, and others contain -Z codegen-backend=llvm?

I've played with it a bit. The generated debuginfo contains both symbols at DW_AT_producer sections:

• rustc version 1.77.0-nightly (bf8716f1c 2023-12-24) with cranelift 0.103.0
• clang LLVM (rustc version 1.77.0-nightly (bf8716f1c 2023-12-24))

Not sure if it implies mixing different codegen backends work. I assume yes?

[DrewRidley]

Hi!

Thanks for getting back to me. I did some more research and it appears you are correct. The documentation was not clear and due to differing guarantees about calling conventions, I sort of just assumed that only a single codegen backend could be used for the entire target.

I did some more testing and found that lto = fat in this condition will cause linker errors and possibly lead to UB. It might make sense for cargo to emit a warning or error in this case. I am not too well informed on the specifics, but I was told that having multiple codegen backends might lead to UB due to different calling conventions between the different backends. If this is the case, what are the plans to mitigate this?

[weihanglo]

I am not too familiar with this area. Let's seek helps from cranelift codegen maintainers.

@bjorn3, what is the risk of mix use of different codegen backends across different crates? Should Cargo ban this kind of usage, or provide a granular control of it and only warn for some cases?

[bjorn3]

Mixing cg_clif and cg_llvm should be completely fine. In fact if you specify cranelift as codegen backend in Cargo.toml you will still use a cg_llvm compiled standard library. I also intend for it to be possible to compile your dependencies with LLVM and full optimizations, but keep your own code compiled with Cranelift for use cases like games where you want the game engine to be as optimized as possible even during development, yet still need quick iteration on changes (like OP's use case) LTO is not supported by cg_clif though and you should get a warning that it can cause linker errors if you try to enable it anyway.

[DrewRidley]

When you say you intend for it to be possible, are you saying that it is not currently possible? The above analysis suggests that the above config already works and produces a binary with assembly from both codegen backends.

I am currently using it in my project and haven't noticed any problems so far, although I think cargo could be improved to show a error message when attempting to use lto = fat in these circumstances.

EDIT:

I just did some more testing and at least in my project, it seems to segfault with this given config and a basic bevy application. The bevy game works until you attempt to build your own plugins and add them to bevy generated with LLVM. There is likely a discrepancy between the call convention from LLVM to Cranelift at this specific point.

Given the expectation that this sort of thing will be supported in some capacity, how will rust resolve ambiguities in these two codegen backends? It sounds a bit out of scope for rustc and there is probably a LOT of nuance to this specific functionality. As a game dev, I fully support your interest in mixing backends like this, but I question how realistic it would be.

I know very little on the topic so any clarification is of course much appreciated. Thank you for your time and interest in my ticket.

[bjorn3]

When you say you intend for it to be possible, are you saying that it is not currently possible?

I mean that it is possible today (minus bugs in edge cases) and not just by accident.

I just did some more testing and at least in my project, it seems to segfault with this given config and a basic bevy application.

Could you please open an issue at https://github.com/rust-lang/rustc_codegen_cranelift with a full reproducer?

[DrewRidley]

I filed an issue here for anyone following: https://github.com/rust-lang/rustc_codegen_cranelift/issues/1449 I will close this now as it seems to be an issue with the codegen itself.

If there is interest, I can file a future issue and PR to include warnings when trying to use LTO with mixed codegens.

[bjorn3]

A warning when trying to mix codegen backends while using LTO would be nice. The warning should likely be done by rustc itself, so please open an issue on the rust repo.

[weihanglo]

According to the above discussion, issues should go to either rust-lang/rustc_codegen_cranelift or rust-lang/rust. I'll close this instead. Thank you.