Float-free libcore (for embedded systems and kernel drivers, among other things)

[emk]

(I was talking to @huonw about embedded Rust the other day, and he suggested I write this up as an RFC issue. I hope this is in the correct place!)

I'm having a ton of fun hacking on kernels in Rust. Rust is a wonderful fit for the problem domain, and the combination of libcore and custom JSON --target specs makes the whole process very ergonomic. But there's one issue that keeps coming up on #rust-osdev: libcore requires floating point, but many otherwise reasonable environments place restrictions on floating point use.

Existing discussions of this issue can be found here:

• Compiling libcore without SSE leads to LLVM ERROR: SSE register return with SSE disabled. If you disable floats in LLVM and rustc, you break the parts of libcore that deal with floats.
• Fixes #26449. A pull request to support libcore without floats, closed without merge. A version of this patch is provided by rust-barebones-kernel, and this patch is frequently recommended on #rust-osdev.
• Are floats really dependency free?. Briefly discusses the fact that libcore depends on floating point.
• Pre-RFC: Dealing with broken floating point. This is a more general discussion of floating point issues, but at one point it proposes #[cfg(float_is_broken)]. Not sure how relevant this is.

  Datum 1: Some otherwise reasonable processors do not support floating point

There's always been a market for embedded processors without an FPU. For the most part, these aren't pathologically weird processors. The standard ARM toolchain supports --fpu=none. Many of the older and/or lower-end ARM chips lack FPUs. For example, the FPU is optional on the Cortex-M4.

Now, I concur (enthusiastically) that not all embedded processors are suitable for Rust. In particular, there are processors where the smallest integer types are u32 and i32, making sizeof(char) == sizeof(uint32_t) == 1 in C, and where uint8_t literally does not exist. There were once quite a few CPUs with 36-bit words. I agree that all these CPUs are all fundamentally unsuitable for Rust, because Rust makes the simplifying decision that the basic integer types are 8, 16, 32 and 64 bits wide, to the immense relief of everybody who programs in Rust.

But CPUs without floating point are a lot more common than CPUs with weird-sized bytes. And the combination of rustc and libcore is an otherwise terrific toolchain for writing low-level code for this family of architecture.

Datum 2: Linux (and many other kernels) forbid floating point to speed up syscalls and interrupts

Another pattern comes up very often:

1. Everybody likes CPUs with a lot of floating point registers, and even a lot of vector floating point registers.
2. Saving all those floating point registers during a syscall or hardware interrupt can be very expensive. You need to save all the registers to switch tasks, of course, but what if you just want to call write or another common syscall?
3. It's entirely possible to write large amounts of kernel code without needing floating point.

These constraints point towards an obvious optimization: If you forbid the use of floating point registers in kernel space, you can handle syscalls and interrupts without having to save the floating point state. This allows you to avoid calling epic instructions like FXSAVE every time you enter kernel space. Yup, FXSAVE stores 512 bytes of data.

Because of these considerations, Linux normally avoids floating point in kernel space. But ARM developers trying to speed up task switching may also do something similar. And this is a very practical issue for people who want to write Linux kernel modules in Rust.

(Note that this also means that LLVM can't use SSE2 instructions for optimizing copies, either! So it's not just a matter of avoiding f32 and f64; you also need to configure your compiler correctly. This has consequences for how we solve this problem, below.)

Possible solutions

Given this background, I'd argue that "libcore without floats" is a fairly well-defined and principled concept, and not just, for example, a rare pathological configuration to support one broken vendor.

There are several different ways that this might be implemented:

1. Make it possible to disable f32 and f64 when building libcore. This avoids tripping over places where the ABI mandates the use of SSE2 registers for floating point, as in https://github.com/rust-lang/rust/issues/26449. The rust-barebones-kernel libcore_nofp.patch shows that this is trivially easy to do.
2. Move f32 and f64 support out of libcore and into a higher-level crate. I don't have a good feel for the tradeoffs here—perhaps it would be good to avoid crate proliferation—but this is one possible workaround.
3. Require support for soft floats in the LLVM & rustc toolchain, even when the platform ABI mandates the use of SSE2 registers. But this is fragile and cumbersome, because it requires maintaining a (custom?) float ABI on platforms even where none exists. And this is currently broken even for x86_64 (https://github.com/rust-lang/rust/issues/26449 again), so it seems like this approach is susceptible to bit rot.
4. Compile libcore with floats and then try to remove them again with LTO. This is hackish, and it requires the developer to leave SSE2 enabled at compilation time, which may allow SSE2-based optimizations to slip in even where f32 and f64 are never mentioned, which will subtly corrupt memory during syscalls and interrupts.
5. Other approaches? I can't think of any, but I'm sure they exist.

What I'd like to see is a situation where people can build things like Linux kernel modules, pure-Rust kernels and (hypothetically) Cortex-M4 (etc.) code without needing to patch libcore. These all seem like great Rust use cases, and easily disabling floating point is (in several cases) the only missing piece.

[comex]

From a quick grep of LLVM, it seems all that flag does is enable/disable the FXSAVE and FXRSTOR instructions in the assembler.

Hmm; does that mean that if you pass -sse, you can't use SSE instructions in inline assembly blocks? Because that sounds somewhat annoying.

[parched]

does that mean that if you pass -sse, you can't use SSE instructions in inline assembly blocks?

No it doesn't apparently (I just tested), but why would you want to use SSE instructions if you have turned it off?

[jdub]

The reason to turn it off is you don't want uncontrolled use of SSE (or, context switch unsafe) instructions, or use of them anywhere near ABI boundaries. Controlled use is fine, and desirable – you're probably going to want fast crypto, vector acceleration, etc. at some point.

[parched]

Well in that case I would turn +sse back on for just that compilation unit then you get the benefit in rust code too.

[jdub]

That'd be pretty uncontrolled. Generally, a few inline functions to safely wrap assembly blocks will suffice.

[carlpaten]

What's the "state of the union" on this particular issue? Do we still need to use libcore_nofp.patch?

[ketsuban]

The state appears to be that according to @japaric (link] and @parched (link) using the new soft-float feature causes floating-point values to be stored in general-purpose registers and manipulated via software implementations. @Amanieu notes that you'll need to recompile compiler-rt since the float ABI has changed, but the standard one will work if you never actually use a floating-point value, so my inference is you'll still need to recompile something, just not necessarily libcore.

[parched]

Well, you shouldn't really recompile anything. 'soft-float' (or equivalent for other targets) should already be set as a feature of the target (or not) which you shouldn't change with 'rustc' codegen options otherwise stuff won't link probably. Obviously if you need to create a new custom target for this then all of 'core'/'std' needs to be compiled as usual with a custom target.

[Mart-Bogdan]

I would like to add, that it's actually possible to use floats inside Linux kernel:

    kernel_fpu_begin();
    ...
    kernel_fpu_end();

But with some restrictions. This function is not rentable, is pretty undocumented and AFAIK would make current thread non-premptable.

The same story for Windows Kernel, thou is more documented. And it's seems it also makes thread non-preemptable (if I understand what "disables all kernel-mode APC delivery" means).

So to sum this up. In C world you could use floats in the kernel if made precautions, but it's encouraged to use it in localized code and disable FPU as soon as possible.

If we are talking about C float-related code could be moved to a separate .c file that is compiled with different options and then linked together with main code.

Not sure how that could be addressed in Rust and with libcore. We could also use linker and move FPU code to separate crate, called using C ABI. But perhaps a better solution could be worked.

P.S. I'm not experienced with this stuff, googled this stuff, and decided to add this info to the current thread as it's not covered in discussion.

[Serentty]

This is a real pain for writing Rust for the Commodore 64 and other 6502 machines. Currently core needs to be patched to disable floating point.

[Qix-]

Hitting this myself today, though not sure if this is a Rust language thing or an LLVM thing. The compiler appears to be emitting fmov instructions on aarch64 in a kernel context, which explicitly doesn't have CPACR_EL1[FPEN] set, which is causing ptr::volatile_write() to fail due to it being emitted in debug builds in the precondition check for pointer alignment.

Is there really no way to shut off floating point register / instruction emission in Rust after 9 years? GCC has had -mgeneral-regs-only for ages.

[Amanieu]

You can use the aarch64-unknown-none-softfloat target to generate AArch64 code which doesn't use the FP registers.

[Qix-]

@Amanieu that doesn't seem to fix it when specifying it in "llvm-target": "aarch64-unknown-none-softfloat" in the target JSON file. I'm still getting an fmov emission.

<_ZN4core3ptr9const_ptr33_$LT$impl$u20$$BP$const$u20$T$GT$13is_aligned_to17h3f6ddebdb141eeffE+32>  fmov  d0, x1

EDIT: Finally found the full list of features in LLVM. I needed to turn off most of the FP features:

"features": "+strict-align,-neon,-fp-armv8,-sm4,-sha2,-sha3,-aes,-crypto,-crc,-rdm,-fp16fml,-sve,-sve2,-sve2-aes,-fptoint",

Thank you @Amanieu :)

[Amanieu]

This is a builtin rustc target, available through rustup.

If you're already using a target json then you can base it on the built-in json for the target:

rustc --print target-spec-json --target aarch64-unknown-none-softfloat -Z unstable-options