SIGILL on PPC32 inside RAND_poll() called by libgit2_sys::init()

[tarcieri]

I am encountering cargo crashing with SIGILL on 32-bit PowerPC 7447A after installing with rustup:

Program received signal SIGILL, Illegal instruction.
0x20d89334 in RAND_poll ()
(gdb) bt
#0  0x20d89334 in RAND_poll ()
#1  0x20d887dc in ssleay_rand_bytes ()
#2  0x20c6e5fc in SSL_CTX_new ()
#3  0x20b42e64 in git_openssl_stream_global_init ()
#4  0x20b291d4 in init_common ()
#5  0x20b29388 in init_once ()
#6  0x20707cdc in __GI___pthread_once (once_control=0x20fe78e8 <_once_init>, init_routine=0x20b29324 <init_once>) at ../nptl/sysdeps/unix/sysv/linux/powerpc/pthread_once.c:86
#7  0x20b29414 in git_libgit2_init ()
#8  0x20b25e1c in std::sync::once::Once::call_once::_$u7b$$u7b$closure$u7d$$u7d$::h4f3f0378b6841f4d ()
#9  0x20df95e4 in std::sync::once::{{impl}}::call_inner () at /buildslave/rust-buildbot/slave/beta-dist-rustc-cross-rustbuild-linux/build/src/libstd/sync/once.rs:288
#10 0x20b25fa0 in libgit2_sys::init::h9ec74b5dead701aa ()
#11 0x20b19840 in git2::config::Config::open_default::h2c6ef11c5f7ed621 ()
#12 0x209841bc in cargo::ops::registry::http_proxy::h8d5082d31cf4d553 ()
#13 0x20984348 in cargo::ops::registry::http_proxy_exists::hd3c3ce97150bd2ac ()
#14 0x2081e254 in cargo::execute::hde538bc4de650fee ()
#15 0x20814dc8 in cargo::call_main_without_stdin::hba58ee9cd44dd247 ()
#16 0x2081de04 in cargo::main::h4a870b5478195f89 ()
#17 0x20e05f30 in std::panicking::try::do_call<fn(),()> () at /buildslave/rust-buildbot/slave/beta-dist-rustc-cross-rustbuild-linux/build/src/libstd/panicking.rs:460
#18 0x20e0f058 in panic_unwind::__rust_maybe_catch_panic () at /buildslave/rust-buildbot/slave/beta-dist-rustc-cross-rustbuild-linux/build/src/libpanic_unwind/lib.rs:98
#19 0x20e06ad8 in try<(),fn()> () at /buildslave/rust-buildbot/slave/beta-dist-rustc-cross-rustbuild-linux/build/src/libstd/panicking.rs:436
#20 catch_unwind<fn(),()> () at /buildslave/rust-buildbot/slave/beta-dist-rustc-cross-rustbuild-linux/build/src/libstd/panic.rs:361
#21 std::rt::lang_start () at /buildslave/rust-buildbot/slave/beta-dist-rustc-cross-rustbuild-linux/build/src/libstd/rt.rs:57
#22 0x20862910 in main ()

This occurs after a SIGILL inside OPENSSL_crypto207_probe(), which appears deliberate as that function is probing CPU capabilities. The OPENSSL_crypto207_probe() handles the SIGILL it causes, but RAND_poll() appears to be using an unsupported instruction which causes cargo to crash.

[tonychain]

Here is a dump of the assembly for RAND_poll:

Dump of assembler code for function RAND_poll:
   0x20bd01e4 <+0>: stwu    r1,-384(r1)
   0x20bd01e8 <+4>: mflr    r0
   0x20bd01ec <+8>: bcl     20,4*cr7+so,0x20bd01f0 <RAND_poll+12>
   0x20bd01f0 <+12>:    stw     r30,368(r1)
   0x20bd01f4 <+16>:    stw     r29,364(r1)
   0x20bd01f8 <+20>:    stw     r0,388(r1)
   0x20bd01fc <+24>:    stw     r21,332(r1)
   0x20bd0200 <+28>:    stw     r22,336(r1)
   0x20bd0204 <+32>:    stw     r23,340(r1)
   0x20bd0208 <+36>:    li      r22,1
   0x20bd020c <+40>:    li      r23,0
   0x20bd0210 <+44>:    stw     r24,344(r1)
   0x20bd0214 <+48>:    stw     r25,348(r1)
   0x20bd0218 <+52>:    mflr    r30
   0x20bd021c <+56>:    stw     r27,356(r1)
   0x20bd0220 <+60>:    stw     r28,360(r1)
   0x20bd0224 <+64>:    li      r28,0
   0x20bd0228 <+68>:    lwz     r0,-16(r30)
   0x20bd022c <+72>:    stw     r31,372(r1)
   0x20bd0230 <+76>:    li      r31,0
   0x20bd0234 <+80>:    stfd    f31,376(r1)
   0x20bd0238 <+84>:    stw     r26,352(r1)
   0x20bd023c <+88>:    add     r30,r0,r30
   0x20bd0240 <+92>:    bl      0x20e2dc88 <getpid@plt>
   0x20bd0244 <+96>:    addi    r29,r1,8
   0x20bd0248 <+100>:   li      r4,0
   0x20bd024c <+104>:   li      r5,264
   0x20bd0250 <+108>:   addi    r24,r1,304
   0x20bd0254 <+112>:   mr      r25,r3
   0x20bd0258 <+116>:   mr      r3,r29
   0x20bd025c <+120>:   bl      0x20e2ddf0 <memset@plt>
   0x20bd0260 <+124>:   lwz     r27,-32764(r30)
   0x20bd0264 <+128>:   addi    r21,r1,272
   0x20bd0268 <+132>:   addi    r27,r27,-4
   0x20bd026c <+136>:   lwzu    r3,4(r27)
   0x20bd0270 <+140>:   li      r4,2304
   0x20bd0274 <+144>:   crclr   4*cr1+eq
   0x20bd0278 <+148>:   bl      0x20e2dc50 <open@plt>
   0x20bd027c <+152>:   mr.     r26,r3
   0x20bd0280 <+156>:   blt     0x20bd0300 <RAND_poll+284>
   0x20bd0284 <+160>:   li      r3,3
   0x20bd0288 <+164>:   mr      r4,r26
   0x20bd028c <+168>:   mr      r5,r29
   0x20bd0290 <+172>:   bl      0x20e2e280 <__fxstat@plt>
   0x20bd0294 <+176>:   cmpwi   cr7,r3,0
   0x20bd0298 <+180>:   bne     cr7,0x20bd02f8 <RAND_poll+276>
   0x20bd029c <+184>:   cmpwi   cr7,r28,0
   0x20bd02a0 <+188>:   beq     cr7,0x20bd02c8 <RAND_poll+228>
   0x20bd02a4 <+192>:   lwz     r9,12(r29)
   0x20bd02a8 <+196>:   lwz     r10,20(r1)
   0x20bd02ac <+200>:   cmpw    cr7,r10,r9
   0x20bd02b0 <+204>:   beq     cr7,0x20bd0490 <RAND_poll+684>
   0x20bd02b4 <+208>:   cmplwi  cr7,r28,2
   0x20bd02b8 <+212>:   bne     cr7,0x20bd02c8 <RAND_poll+228>
   0x20bd02bc <+216>:   lwz     r10,108(r1)
   0x20bd02c0 <+220>:   cmpw    cr7,r9,r10
   0x20bd02c4 <+224>:   beq     cr7,0x20bd0460 <RAND_poll+636>
   0x20bd02c8 <+228>:   mr      r3,r24
   0x20bd02cc <+232>:   li      r4,1
   0x20bd02d0 <+236>:   stw     r26,304(r1)
   0x20bd02d4 <+240>:   sth     r22,308(r1)
   0x20bd02d8 <+244>:   li      r5,10
   0x20bd02dc <+248>:   sth     r23,310(r1)
   0x20bd02e0 <+252>:   bl      0x20e2e198 <poll@plt>
   0x20bd02e4 <+256>:   cmpwi   cr7,r3,0
   0x20bd02e8 <+260>:   blt     cr7,0x20bd02f8 <RAND_poll+276>
   0x20bd02ec <+264>:   lhz     r9,310(r1)
   0x20bd02f0 <+268>:   andi.   r10,r9,1
   0x20bd02f4 <+272>:   bne     0x20bd0440 <RAND_poll+604>
   0x20bd02f8 <+276>:   mr      r3,r26
   0x20bd02fc <+280>:   bl      0x20e2e0e8 <close@plt>
   0x20bd0300 <+284>:   cmpwi   cr7,r28,2
   0x20bd0304 <+288>:   addi    r28,r28,1
   0x20bd0308 <+292>:   beq     cr7,0x20bd03e0 <RAND_poll+508>
   0x20bd030c <+296>:   cmpwi   cr7,r31,31
   0x20bd0310 <+300>:   addi    r29,r29,88
   0x20bd0314 <+304>:   ble     cr7,0x20bd026c <RAND_poll+136>
   0x20bd0318 <+308>:   addi    r28,r1,272
   0x20bd031c <+312>:   srawi   r10,r31,31
   0x20bd0320 <+316>:   stw     r10,312(r1)
   0x20bd0324 <+320>:   stw     r31,316(r1)
   0x20bd0328 <+324>:   mr      r3,r28
   0x20bd032c <+328>:   li      r4,32
   0x20bd0330 <+332>:   lfd     f0,312(r1)
=> 0x20bd0334 <+336>:   fcfid   f1,f0
   0x20bd0338 <+340>:   bl      0x20b065b0 <RAND_add>
   0x20bd033c <+344>:   mr      r3,r28
   0x20bd0340 <+348>:   mr      r4,r31
   0x20bd0344 <+352>:   bl      0x20acb960 <OPENSSL_cleanse>
   0x20bd0348 <+356>:   lwz     r9,-32760(r30)
   0x20bd034c <+360>:   mr      r3,r24
   0x20bd0350 <+364>:   li      r4,4
   0x20bd0354 <+368>:   stw     r25,304(r1)
   0x20bd0358 <+372>:   lfs     f31,0(r9)
   0x20bd035c <+376>:   fmr     f1,f31
   0x20bd0360 <+380>:   bl      0x20b065b0 <RAND_add>
   0x20bd0364 <+384>:   bl      0x20e2e088 <getuid@plt>
   0x20bd0368 <+388>:   fmr     f1,f31
   0x20bd036c <+392>:   li      r4,4
   0x20bd0370 <+396>:   stw     r3,304(r1)
   0x20bd0374 <+400>:   mr      r3,r24
   0x20bd0378 <+404>:   bl      0x20b065b0 <RAND_add>
   0x20bd037c <+408>:   li      r3,0
   0x20bd0380 <+412>:   bl      0x20e2e108 <time@plt>
   0x20bd0384 <+416>:   fmr     f1,f31
   0x20bd0388 <+420>:   li      r4,4
   0x20bd038c <+424>:   stw     r3,304(r1)
   0x20bd0390 <+428>:   mr      r3,r24
   0x20bd0394 <+432>:   bl      0x20b065b0 <RAND_add>
   0x20bd0398 <+436>:   lwz     r0,388(r1)
   0x20bd039c <+440>:   lwz     r21,332(r1)
   0x20bd03a0 <+444>:   li      r3,1
   0x20bd03a4 <+448>:   lwz     r22,336(r1)
   0x20bd03a8 <+452>:   lwz     r23,340(r1)
   0x20bd03ac <+456>:   mtlr    r0
   0x20bd03b0 <+460>:   lwz     r24,344(r1)
   0x20bd03b4 <+464>:   lwz     r25,348(r1)
   0x20bd03b8 <+468>:   lwz     r26,352(r1)
   0x20bd03bc <+472>:   lwz     r27,356(r1)
   0x20bd03c0 <+476>:   lwz     r28,360(r1)
   0x20bd03c4 <+480>:   lwz     r29,364(r1)
   0x20bd03c8 <+484>:   lwz     r30,368(r1)
   0x20bd03cc <+488>:   lwz     r31,372(r1)
   0x20bd03d0 <+492>:   lfd     f31,376(r1)
   0x20bd03d4 <+496>:   addi    r1,r1,384
   0x20bd03d8 <+500>:   blr
   0x20bd03dc <+504>:   nop
   0x20bd03e0 <+508>:   lwz     r29,-32768(r30)
   0x20bd03e4 <+512>:   lwz     r3,0(r29)
   0x20bd03e8 <+516>:   cmpwi   cr7,r3,0
   0x20bd03ec <+520>:   beq     cr7,0x20bd042c <RAND_poll+584>
   0x20bd03f0 <+524>:   cmpwi   cr7,r31,31
   0x20bd03f4 <+528>:   addi    r28,r1,272
   0x20bd03f8 <+532>:   ble     cr7,0x20bd0404 <RAND_poll+544>
   0x20bd03fc <+536>:   b       0x20bd031c <RAND_poll+312>
   0x20bd0400 <+540>:   bgt     cr6,0x20bd031c <RAND_poll+312>
   0x20bd0404 <+544>:   add     r4,r28,r31
   0x20bd0408 <+548>:   subfic  r5,r31,32
   0x20bd040c <+552>:   bl      0x20b067a0 <RAND_query_egd_bytes>
   0x20bd0410 <+556>:   cmpwi   r3,0
   0x20bd0414 <+560>:   ble     0x20bd041c <RAND_poll+568>
   0x20bd0418 <+564>:   add     r31,r31,r3
   0x20bd041c <+568>:   lwzu    r3,4(r29)
   0x20bd0420 <+572>:   cmpwi   cr6,r31,31
   0x20bd0424 <+576>:   cmpwi   cr7,r3,0
   0x20bd0428 <+580>:   bne     cr7,0x20bd0400 <RAND_poll+540>
   0x20bd042c <+584>:   cmpwi   cr7,r31,0
   0x20bd0430 <+588>:   beq     cr7,0x20bd0348 <RAND_poll+356>
   0x20bd0434 <+592>:   b       0x20bd0318 <RAND_poll+308>
   0x20bd0438 <+596>:   nop
   0x20bd043c <+600>:   nop
   0x20bd0440 <+604>:   mr      r3,r26
   0x20bd0444 <+608>:   add     r4,r21,r31
   0x20bd0448 <+612>:   subfic  r5,r31,32
   0x20bd044c <+616>:   bl      0x20e2de80 <read@plt>
   0x20bd0450 <+620>:   cmpwi   r3,0
   0x20bd0454 <+624>:   ble     0x20bd02f8 <RAND_poll+276>
   0x20bd0458 <+628>:   add     r31,r31,r3
   0x20bd045c <+632>:   b       0x20bd02f8 <RAND_poll+276>
   0x20bd0460 <+636>:   lwz     r10,184(r1)
   0x20bd0464 <+640>:   lwz     r9,96(r1)
   0x20bd0468 <+644>:   cmpw    cr7,r9,r10
   0x20bd046c <+648>:   bne     cr7,0x20bd02c8 <RAND_poll+228>
   0x20bd0470 <+652>:   lwz     r10,188(r1)
   0x20bd0474 <+656>:   lwz     r9,100(r1)
   0x20bd0478 <+660>:   cmpw    cr7,r9,r10
   0x20bd047c <+664>:   bne     cr7,0x20bd02c8 <RAND_poll+228>
   0x20bd0480 <+668>:   b       0x20bd02f8 <RAND_poll+276>
   0x20bd0484 <+672>:   nop
   0x20bd0488 <+676>:   nop
   0x20bd048c <+680>:   nop
   0x20bd0490 <+684>:   lwz     r8,0(r29)
   0x20bd0494 <+688>:   lwz     r10,8(r1)
   0x20bd0498 <+692>:   cmpw    cr7,r10,r8
   0x20bd049c <+696>:   bne     cr7,0x20bd02b4 <RAND_poll+208>
   0x20bd04a0 <+700>:   lwz     r8,4(r29)
   0x20bd04a4 <+704>:   lwz     r10,12(r1)
   0x20bd04a8 <+708>:   cmpw    cr7,r10,r8
   0x20bd04ac <+712>:   bne     cr7,0x20bd02b4 <RAND_poll+208>
   0x20bd04b0 <+716>:   b       0x20bd02f8 <RAND_poll+276>
End of assembler dump.

[alexcrichton]

cc @cuviper are you familiar enough with powerpc to know if we should be passing some -march flag or something like that when compiling powerpc?

[tonychain]

According to some random thread I found:

https://gcc.gnu.org/ml/gcc-patches/2012-04/msg00697.html

It was brought to my attention that when I rewrote the floating point conversion operations for power7, I did not notice that the power4 and 970 powerpc's actually support the FCFID (floating point convert) instruciton in 32-bit mode.

Is it possible this is being built on a PPC970 (a.k.a. "G5") in 32-bit mode? In which case, the FCFID instruction will be available there, but not on other PPC32 CPUs

[cuviper]

The cross-compiling toolchain was configured with a pretty conservative default, effectively -march=power4 -mtune=power6, which permits fcfid. But it seems gcc didn't output fcfid for 32-bit power4 until 4.8: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=52775

I'm not certain about IBM history, but PowerPC is a separate line from the POWER series, so it wouldn't have the same ISA as POWER4. PowerPC 7447A is the G4 series, and AFAICT only G5 implemented fcfid.

[alexcrichton]

Oh awesome, thanks for the information! @cuviper do you think this is a case of "this cpu is unsupported" or do you think this is "we should pass a flag to generate an even older ISA"?

[cuviper]

Seems to me that running on even older ppc hardware is a pretty niche thing to do, but I won't object if someone wants to try to get it working. See dist-powerpc-linux and its entry in the README, and please CC me if you do make changes to this.

[tonychain]

It seems like unless this is fixed (and provided I'm actually understanding what's going on), PPC32 cargo will only work on PPC64 CPUs running in 32-bit mode, as I don't believe this instruction is supported on any natively 32-bit PPC CPUs.

[cuviper]

I hope this will be fixed by rust-lang/rust#41080, but I don't have the hardware to test it.

[tarcieri]

Awesome! I can give it a try when it lands

[tarcieri]

So I gave this a try... I'm on this version of rustc:

rustc 1.16.0 (30cf806ef 2017-03-10)

I'm still getting the SIGILL in Cargo at FCFID:

   0x2065132c <+328>:   li      r4,32
   0x20651330 <+332>:   lfd     f0,312(r1)
=> 0x20651334 <+336>:   fcfid   f1,f0
   0x20651338 <+340>:   bl      0x205875b0 <RAND_add>
   0x2065133c <+344>:   mr      r3,r28

But this seems to have broken rustup as well:

$ rustup update
info: syncing channel updates for 'stable-powerpc-unknown-linux-gnu'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' to '/home/tony/.rustup/tmp/qo074736oqcxuo4q_file'
info: syncing channel updates for 'nightly-powerpc-unknown-linux-gnu'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-nightly.toml.sha256' to '/home/tony/.rustup/tmp/p_l7w0k7vkeiphj7_file'
info: checking for self-updates
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '/tmp/rustup-update.ZxVWTQGeg7AT/release-stable.toml'
info: caused by: error during download
info: caused by: [1] Unsupported protocol (Protocol "https" not supported or disabled in libcurl)

[cuviper]

The PPC build change is so recent that it should only show up on the nightly builds so far. I have no idea about your rustup errors, but it looks like a different issue.

[tonychain]

Okay, weird, I couldn't get rustup.rs to work, but rustup.sh worked like a charm and now cargo is working with nightly, thanks!

[alexcrichton]

Awesome, thanks for confirming!