Built binaries crash with illegal hardware instruction on BCM2708 cpu (raspberry pi)

[lilydjwg]

Cross build a "Hello World" program generated by cargo, and it fails to run.

I use the rustc and rust-std-nightly-arm-unknown-linux-gnueabihf.tar.gz from https://static.rust-lang.org/dist/

>>> rustc --version
rustc 1.8.0-nightly (57c357d89 2016-02-16)

The crash in gdb:

Program received signal SIGILL, Illegal instruction.
0x7f559c5c in jemalloc_constructor ()
(gdb) bt
#0  0x7f559c5c in jemalloc_constructor ()
#1  0x7f5879f0 in __libc_csu_init ()
#2  0xb6e3ab74 in __libc_start_main () from /usr/lib/libc.so.6
#3  0x7f559d94 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Dump of assembler code for function jemalloc_constructor:
   0x7f559c50 <+0>:     ldr     r3, [pc, #212]  ; (0x7f559d28 <jemalloc_constructor+216>)
   0x7f559c52 <+2>:     push    {r4, lr}
   0x7f559c54 <+4>:     add     r3, pc
   0x7f559c56 <+6>:     ldr     r4, [pc, #212]  ; (0x7f559d2c <jemalloc_constructor+220>)
   0x7f559c58 <+8>:     ldrb    r0, [r3, #0]
   0x7f559c5a <+10>:    add     r4, pc
=> 0x7f559c5c <+12>:    cbz     r0, 0x7f559c68 <jemalloc_constructor+24>
   0x7f559c5e <+14>:    ldr     r1, [pc, #208]  ; (0x7f559d30 <jemalloc_constructor+224>)
   0x7f559c60 <+16>:    add     r1, pc
   0x7f559c62 <+18>:    ldr     r2, [r1, #0]

/proc/cpuinfo:

processor       : 0
model name      : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS        : 2.00
Features        : half thumb fastmult vfp edsp java tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xb76
CPU revision    : 7

Hardware        : BCM2708
Revision        : 000d
Serial          : 000000000d4fede5

The gcc toolchain used to link the program can build runnable C program on this pi.

[MagaTailor]

You didn't say enough about your rustc (host, target) - I suspect you picked up an armv7 version.

You can try using armv6 crates from RustBuild or compile for the soft-float (gnueabi) target.

[MagaTailor]

Wait, I forgot Rust's SIGILL doesn't have to come from an actual illegal instruction. It's probably a sign of some unexpected crash related to jemalloc. Try a version of rustc that was built without jemalloc or add this to your program to use the system allocator:

#![feature(alloc_system)]
extern crate alloc_system ;

[alexcrichton]

Looks like the faulting instruction is cbz (thanks for the disassembly!) which according to ARM is only a 16-bit thumb instruction. I guess your CPU doesn't have 16-bit thumb instructions?

I guess this means that gcc emits these instructions by default, and I'm not sure if LLVM does (or if we should disable a feature here and there). The suggestion of trying alloc_system will likely definitely help here though in diagnosing!

@petevine as mentioned in the report the arm-unknown-linux-gnueabihf is being used (not armv7). Also the illegal instruction looks like it's not coming from Rust at all (as evidenced by the disassembly).

@lilydjwg does a simple "hello world" exhibit the crash you're seeing here? Or is the program at hand more complicated?

[MagaTailor]

Before posting the second time I'd checked the cbz instruction's presence in armv6 (16-bit is the feature of thumb that is supposed to make the code more compact) and the pasted cpu features clearly indicate thumb is supported.

@alexcrichton

What's the mode gcc is using on the buildbot? (like armv6, thumb). That's probably the source of this error cause the target triple was originally also meant to work on armv7.

[lilydjwg]

@petevine my rustc is from the same place as rust-std, all with the same version. There's an "arm-unknown-linux-gnueabi" version in the std library and I tried that, it failed to link. I suppose it's because my toolchain is gnueabihf.

@alexcrichton the hello world program crashes. A page from Google says "cbz" is from thumb2, and cpuinfo indicates my CPU doesn't support that.

Adding

#![feature(alloc_system)]
extern crate alloc_system;

to my test crate and it works! So it's only jemalloc is using those "illegal" instructions?

[MagaTailor]

For some reason I was convinced Rpi had thumb2 (Rpi2 has) as cpuinfo doesn't differentiate between thumb and thumb2.

You could also try those unofficial armv6 crates to see if jemalloc had been built without thumb2. However, I'm pretty sure it's not going to offer any performance benefit on that old cpu (and 512M of ram) so you're better off without jemalloc.

[lilydjwg]

@petevine there are several versions of rpi (and I don't know which is the one I have...).

I can't try the unofficial armv6 crates becausse I don't have an x86_64 rustc from the same commit and they don't provide one. My pi is not only very slow at compiling things but also has too little disk space...

[MagaTailor]

You definitely have the original Rpi (or Zero) as it doesn't support T2.

If you found an unofficial armv6 nightly from the same day and it had library hashes matching your x86_64 ones you should be able to use them for cross-compilation.

Or just wait until buildbot gets fixed.

[alexcrichton]

Thanks for the info @lilydjwg! Looks like our compilers for ARM are defaulting to armv7 and/or thumb2 things by default. This differs from what clang does I believe, so I've submitted https://github.com/rust-lang/rust/pull/31800 to hopefully tell gcc to emit the right kind of code.

[lilydjwg]

@petevine I managed to get it cross compiled, but then it turned out that armv6 build does not have jemalloc:

error: can't find crate for `alloc_jemalloc`

@alexcrichton will there be built crates for me to test with? Or how can I build them without build rustc and all the stages?

[alexcrichton]

Unfortunately there's no great and easy way to do that right now

[MagaTailor]

Thanks @lilydjwg , I didn't know @warricksothr had disabled jemalloc in his builds too.

[michaelherger]

Hi there - what rust environment would I have to have in order to get this fix? I installed rust using rustup, and it's currently on rustc 1.17.0 (56124baa9 2017-04-24). Should that be good enough?

I'm asking because I'm seeing "illegal instruction" errors trying to run a arm-unknown-linux-musleabihf build on a Pi1.

[hmvp]

I thought I was hitting this with the cnbz instruction but it turns out that arm-linux-gnueabihf-gcc on Ubuntu produces armv7 binaries. see: #38570