SIGILL when creating wallet on CPU without AVX support

iridescent86 commented 4 years ago

Description

Running lighthouse account wallet create --name validators --passphrase-file validators.pass fails with SIGILL when CPU has no AVX support.

Version

lighthouse 0.1.2 built from master branch, commit ce10db15da0db4cbe76b96b58ccd1b40e39ed124
rustc 1.43.1 (8d69840ab 2020-05-04)

Present Behaviour

When running the wallet create command on a CPU that does not has AVX instructions set support, it fails and raises SIGILL:

$ ./lighthouse/target/release/lighthouse account wallet create --name validators --passphrase-file validators.pass
Illegal instruction

GDB output:

$ gdb ./lighthouse/target/release/lighthouse
(gdb) r account wallet create --name validators --passphrase-file validators.pass
Starting program: /home/admin/lighthouse/target/release/lighthouse account wallet create --name validators --passphrase-file validators.pass
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff72c9700 (LWP 31454)]
[New Thread 0x7ffff70c8700 (LWP 31455)]

Thread 1 "lighthouse" received signal SIGILL, Illegal instruction.
0x000055555613be01 in rust_crypto_aesni_setup_working_key_128 ()

(gdb) where
#0  0x000055555613be01 in rust_crypto_aesni_setup_working_key_128 ()
#1  0x000055555613bd97 in rust_crypto_aesni_setup_working_key_128 ()
#2  0x000055555613c311 in crypto::aesni::setup_working_key_aesni_128 ()
#3  0x000055555612fc86 in crypto::aes::ctr ()
#4  0x000055555612c344 in eth2_keystore::keystore::encrypt ()
#5  0x00005555560e35d1 in eth2_wallet::wallet::WalletBuilder::build ()
#6  0x00005555560c2262 in eth2_wallet_manager::wallet_manager::WalletManager::create_wallet ()
#7  0x0000555556061e3a in account_manager::wallet::create::cli_run ()
#8  0x000055555605830f in account_manager::wallet::cli_run ()
#9  0x0000555555ba66e1 in account_manager::run ()
#10 0x0000555555dbab45 in lighthouse::run ()
#11 0x0000555555db3aed in lighthouse::main ()
#12 0x0000555555ab05e3 in std::rt::lang_start::{{closure}} ()
#13 0x00005555569d0b03 in std::rt::lang_start_internal::{{closure}} () at src/libstd/rt.rs:52
#14 std::panicking::try::do_call () at src/libstd/panicking.rs:303
#15 0x00005555569d74b7 in __rust_maybe_catch_panic () at src/libpanic_unwind/lib.rs:86
#16 0x00005555569d162c in std::panicking::try () at src/libstd/panicking.rs:281
#17 std::panic::catch_unwind () at src/libstd/panic.rs:394
#18 std::rt::lang_start_internal () at src/libstd/rt.rs:51
#19 0x0000555555dc0272 in main ()

(gdb) disassemble 0x000055555613be01
Dump of assembler code for function rust_crypto_aesni_setup_working_key_128:
   0x000055555613bd80 <+0>:     movdqu (%rdi),%xmm1
   0x000055555613bd84 <+4>:     movdqu %xmm1,(%rsi)
   0x000055555613bd88 <+8>:     add    $0x10,%rsi
   0x000055555613bd8c <+12>:    aeskeygenassist $0x1,%xmm1,%xmm2
   0x000055555613bd92 <+18>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bd97 <+23>:    aeskeygenassist $0x2,%xmm1,%xmm2
   0x000055555613bd9d <+29>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bda2 <+34>:    aeskeygenassist $0x4,%xmm1,%xmm2
   0x000055555613bda8 <+40>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdad <+45>:    aeskeygenassist $0x8,%xmm1,%xmm2
   0x000055555613bdb3 <+51>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdb8 <+56>:    aeskeygenassist $0x10,%xmm1,%xmm2
   0x000055555613bdbe <+62>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdc3 <+67>:    aeskeygenassist $0x20,%xmm1,%xmm2
   0x000055555613bdc9 <+73>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdce <+78>:    aeskeygenassist $0x40,%xmm1,%xmm2
   0x000055555613bdd4 <+84>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdd9 <+89>:    aeskeygenassist $0x80,%xmm1,%xmm2
   0x000055555613bddf <+95>:    callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bde4 <+100>:   aeskeygenassist $0x1b,%xmm1,%xmm2
   0x000055555613bdea <+106>:   callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdef <+111>:   aeskeygenassist $0x36,%xmm1,%xmm2
   0x000055555613bdf5 <+117>:   callq  0x55555613bdfc <rust_crypto_aesni_setup_working_key_128+124>
   0x000055555613bdfa <+122>:   jmp    0x55555613be29 <rust_crypto_aesni_setup_working_key_128+169>
   0x000055555613bdfc <+124>:   pshufd $0xff,%xmm2,%xmm2
=> 0x000055555613be01 <+129>:   vpslldq $0x4,%xmm1,%xmm3
   0x000055555613be06 <+134>:   pxor   %xmm3,%xmm1
   0x000055555613be0a <+138>:   vpslldq $0x4,%xmm1,%xmm3
   0x000055555613be0f <+143>:   pxor   %xmm3,%xmm1
   0x000055555613be13 <+147>:   vpslldq $0x4,%xmm1,%xmm3
   0x000055555613be18 <+152>:   pxor   %xmm3,%xmm1
   0x000055555613be1c <+156>:   pxor   %xmm2,%xmm1
   0x000055555613be20 <+160>:   movdqu %xmm1,(%rsi)
   0x000055555613be24 <+164>:   add    $0x10,%rsi
   0x000055555613be28 <+168>:   retq
   0x000055555613be29 <+169>:   retq
End of assembler dump.

/proc/cpuinfo on the machine on which issue was reproduced:

$ cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 95
model name      : Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
stepping        : 1
microcode       : 0x1
cpu MHz         : 2200.000
cache size      : 2048 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand hypervisor lahf_lm 3dnowprefetch cpuid_fault pti ssbd ibrs ibpb fsgsbase tsc_adjust smep erms mpx rdseed smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves arat umip md_clear
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4400.00
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

On this machine, the issue was reproduced on a KVM based VM as well as on the bare metal.
Only the wallet appears to be affected, the beacon node and the validator client were working fine on that server.

Expected Behaviour

Wallet should be created successfuly as it is the case when testing on another machine with AVX support:

$ ./lighthouse/target/release/lighthouse account wallet create --name validators --passphrase-file validators.pass
Your wallet's 12-word BIP-39 mnemonic is:

        (output omitted)

This mnemonic can be used to fully restore your wallet, should
you lose the JSON file or your password.

It is very important that you DO NOT SHARE this mnemonic as it will
reveal the private keys of all validators and keys generated with
this wallet. That would be catastrophic.

It is also important to store a backup of this mnemonic so you can
recover your private keys in the case of data loss. Writing it on
a piece of paper and storing it in a safe place would be prudent.

Your wallet's UUID is:

        38ceb5f4-23da-4033-b961-54859354e485

You do not need to backup your UUID or keep it secret.

Steps to resolve

Root cause appears to be https://github.com/DaGenix/rust-crypto/issues/391 which was never fixed. Unfortunately there does not seem to be much that can be done at the library level as it appears to no longer be maintained.

paulhauner commented 4 years ago

Thanks for the detailed report @iridescent86. This does look problematic.

We're planning to move away from rust-crypto in the coming week, I hope this will fix this issue as well.

For now, I'll mark this as blocked on #1245. Once it's resolved we can come back to this issue and ensure it works.

paulhauner commented 4 years ago

We have replaced rust-crypto so this issue has been addressed.

sigp / lighthouse