os would always hang after running cpl/idt/memcmp

[xiaonan-INTC]

When run cpl/idt/memcmp, os would always hang after a few minutes, and need to hardware-reboot by manual. Is that normal?

[jovanbulck]

Thank you for reporting, no that doesn't sound normal..

For cpl/idt, these are "selftests" that try out the APIC and IVT reconfigurations. If these hang you machine, I suspect you didn't boot with the correct kernel command line options and get an #GP or kernel panic as a result(?) Please double check this first. If the issue persists, please post full system info and esp the output of the kernel log here (/var/log/kern.log may still have the info up to the crash before the hard reboot)

For memcmp, it seems this example is indeed currently broken (also #50 ), marking this as a bug. I'm still in the process of setting up SGX-Step on a newer machine w Ubuntu 22.04, so I'll try to look into fixing this example then as well (which may probably only be later this week or next week though).

[xiaonan-INTC]

Thanks for the analysis, jovanbulck,

Some update:

For os hang: I set up a Skylake with Ubuntu 20.04, it seems the OS hang disappeared now. I will track it.

For running error: foreshadow/: ... [main.c] prefetching enclave secret (EENTER/EEXIT)... [main.c] extracting secret from L1 cache.. Illegal instruction

idt/: ... [main.c] Triggering ring3 APIC timer interrupts.. [file.c] reading buffer from '/dev/cpu/1/msr' (size=8) [apic.c] established local memory mapping for APIC_BASE=0xfee00000 at 0x7fe0605e7000 [apic.c] APIC_ID=2000000; LVTT=2d; TDCR=0 [apic.c] APIC timer one-shot mode with division 2 (lvtt=2d/tdcr=0) it will hang here

memcmp/: ... [attacker] steps=1; guess='***'[main.c] assertion 'pwd_len < MAX_LEN' failed: Operation not permitted Aborted

[jovanbulck]

Thanks for the updates, some replies below:

For os hang: I set up a Skylake with Ubuntu 20.04, it seems the OS hang disappeared now. I will track it.

Okay, good to hear it disappeared! I'm wondering, can you clarify which OS did you try before when it hanged? Was it a more recent Linux kernel (eg Ubuntu 22.04)? I'm suspecting there may be issues with how more recent Linux kernels react to SGX-Step hijacking the timer etc..

For running error: foreshadow/: ... [main.c] prefetching enclave secret (EENTER/EEXIT)... [main.c] extracting secret from L1 cache.. Illegal instruction

This is probably because your machine doesn't support TSX. You can try to set USE_TSX=0 in main.c and foreshadow.c

Also note that app/foreshadow is a highly unoptimized proof-of-concept, only showcasing the general concept (and suffering from a low success rate).

I should probably better document this and consider releasing a more robust open-source Foreshadow PoC demonstrator when I find time..

idt/: ... [main.c] Triggering ring3 APIC timer interrupts.. [file.c] reading buffer from '/dev/cpu/1/msr' (size=8) [apic.c] established local memory mapping for APIC_BASE=0xfee00000 at 0x7fe0605e7000 [apic.c] APIC_ID=2000000; LVTT=2d; TDCR=0 [apic.c] APIC timer one-shot mode with division 2 (lvtt=2d/tdcr=0) it will hang here

Hm, that means the memory-mapped APIC register writes crash the machine. Are you sure you booted the kernel with nox2apic as documented? If not, you'll hang the machine when using MMIO APIC registers (as documented in Intel SDM).

memcmp/: ... [attacker] steps=1; guess='***'[main.c] assertion 'pwd_len < MAX_LEN' failed: Operation not permitted Aborted

Thanks, I confirmed that memcmp indeed doesn't work anymore on a more recent machine, so I'll need to look into this and fix it.

[xiaonan-INTC]

Thanks for the updates, some replies below:

For os hang: I set up a Skylake with Ubuntu 20.04, it seems the OS hang disappeared now. I will track it.

Okay, good to hear it disappeared! I'm wondering, can you clarify which OS did you try before when it hanged? Was it a more recent Linux kernel (eg Ubuntu 22.04)? I'm suspecting there may be issues with how more recent Linux kernels react to SGX-Step hijacking the timer etc..

For running error: foreshadow/: ... [main.c] prefetching enclave secret (EENTER/EEXIT)... [main.c] extracting secret from L1 cache.. Illegal instruction

This is probably because your machine doesn't support TSX. You can try to set USE_TSX=0 in main.c and foreshadow.c

Also note that app/foreshadow is a highly unoptimized proof-of-concept, only showcasing the general concept (and suffering from a low success rate).

I should probably better document this and consider releasing a more robust open-source Foreshadow PoC demonstrator when I find time..

idt/: ... [main.c] Triggering ring3 APIC timer interrupts.. [file.c] reading buffer from '/dev/cpu/1/msr' (size=8) [apic.c] established local memory mapping for APIC_BASE=0xfee00000 at 0x7fe0605e7000 [apic.c] APIC_ID=2000000; LVTT=2d; TDCR=0 [apic.c] APIC timer one-shot mode with division 2 (lvtt=2d/tdcr=0) it will hang here

Hm, that means the memory-mapped APIC register writes crash the machine. Are you sure you booted the kernel with nox2apic as documented? If not, you'll hang the machine when using MMIO APIC registers (as documented in Intel SDM).

memcmp/: ... [attacker] steps=1; guess='***'[main.c] assertion 'pwd_len < MAX_LEN' failed: Operation not permitted Aborted

Thanks, I confirmed that memcmp indeed doesn't work anymore on a more recent machine, so I'll need to look into this and fix it.

Thanks @jovanbulck,

For os hang: Both are Ubuntu 20.04 LTS, 5.4.0-26-generic. The OS would not hang now, but it always pop up a dialog saying “System program problem detected”.

For running error of foreshadow: Yes, that CPU(i3-7300) does not support TSX, I annotated as yours and it could run, but still had errors as below: [main.c] prefetching enclave secret (EENTER/EEXIT)... [main.c] extracting secret from L1 cache.. [main.c] verifying and destroying enclave secret.. shadow[ 0]=0x00; enclave[ 0]=0x8d shadow[ 1]=0x00; enclave[ 1]=0x9e shadow[ 2]=0x00; enclave[ 2]=0x92 shadow[ 3]=0x00; enclave[ 3]=0x63 ... shadow[60]=0x00; enclave[60]=0x8b shadow[61]=0x00; enclave[61]=0x3b shadow[62]=0x00; enclave[62]=0x9e shadow[63]=0x00; enclave[63]=0xde [foreshadow.c] [FAIL] Foreshadow missed 64 bytes out of 64 :/

For running error of idt: I re-run and the error dasappeared.

[jovanbulck]

Thanks @jovanbulck,

For os hang: Both are Ubuntu 20.04 LTS, 5.4.0-26-generic. The OS would not hang now, but it always pop up a dialog saying “System program problem detected”.

Okay, thanks for clarifying. It is not abornormal when using SGX-Step to get such warnings as it interferes deeply with the kenrel operations.. When you get a hang or want to know what's going on it's best to check the kernel logs for warnings/panics using dmesg (or /var/log/kern.log after reboot).

For running error of foreshadow: Yes, that CPU(i3-7300) does not support TSX, I annotated as yours and it could run, but still had errors as below: [main.c] prefetching enclave secret (EENTER/EEXIT)... [main.c] extracting secret from L1 cache.. [main.c] verifying and destroying enclave secret.. shadow[ 0]=0x00; enclave[ 0]=0x8d shadow[ 1]=0x00; enclave[ 1]=0x9e shadow[ 2]=0x00; enclave[ 2]=0x92 shadow[ 3]=0x00; enclave[ 3]=0x63 ... shadow[60]=0x00; enclave[60]=0x8b shadow[61]=0x00; enclave[61]=0x3b shadow[62]=0x00; enclave[62]=0x9e shadow[63]=0x00; enclave[63]=0xde [foreshadow.c] [FAIL] Foreshadow missed 64 bytes out of 64 :/

Okay, glad to hear it runs now! When the Foreshadow demonstrator consistently recovers 0x00 bytes, that likely means you are running with patched microcode which clears L1D on enclave EENTER/EEXIT. Hence, you can try to run with older microcode (eg using dis_ucode_ldr Linux boot option), or if that's not possible I added an option a while ago to try out Foreshadow from a "simulated" enclave that is not hardware-protected by SGX but simply a different C source file in the same address space, see the option in main.c:

SIM_ENCLAVE         0       /* PoC without EENTER to demonstrate FS on CPUs with patched ucode

For running error of idt: I re-run and the error dasappeared.

Great to hear it works now! For what it's worth, looking at the log above, it appears the APIC was configured in one-shot at the start of the failed run, so perhaps something goes wrong then.. (ie on normal application exit, SGX-Step takes care to restore the APIC in deadline mode, as expected by Linux kernel, but if the app crashes, it doesn't get a chance to still restore the APIC timer, so perhaps that led to the next runs crashing somehow.)

Hope it helps!

[xiaonan-INTC]

For running error of foreshadow：

I tried adding dis_ucode_ldr to /etc/default/grub and setting #define SIM_ENCLAVE 1, still failed as below: shadow[ 0]=0x67; enclave[ 0]=0x67 shadow[ 1]=0xc6; enclave[ 1]=0xc6 shadow[ 2]=0x69; enclave[ 2]=0x69 shadow[ 3]=0x73; enclave[ 3]=0x73 shadow[ 4]=0x51; enclave[ 4]=0x51 shadow[ 5]=0x00; enclave[ 5]=0xff shadow[ 6]=0x4a; enclave[ 6]=0x4a shadow[ 7]=0x00; enclave[ 7]=0xec shadow[ 8]=0x29; enclave[ 8]=0x29 shadow[ 9]=0xcd; enclave[ 9]=0xcd shadow[10]=0xba; enclave[10]=0xba shadow[11]=0xab; enclave[11]=0xab shadow[12]=0x00; enclave[12]=0xf2 shadow[13]=0x00; enclave[13]=0xfb shadow[14]=0x00; enclave[14]=0xe3 shadow[15]=0x46; enclave[15]=0x46 shadow[16]=0x7c; enclave[16]=0x7c shadow[17]=0x00; enclave[17]=0xc2 shadow[18]=0x00; enclave[18]=0x54 shadow[19]=0x00; enclave[19]=0xf8 shadow[20]=0x1b; enclave[20]=0x1b shadow[21]=0xe8; enclave[21]=0xe8 shadow[22]=0x00; enclave[22]=0xe7 shadow[23]=0x8d; enclave[23]=0x8d shadow[24]=0x76; enclave[24]=0x76 shadow[25]=0x5a; enclave[25]=0x5a shadow[26]=0x2e; enclave[26]=0x2e shadow[27]=0x63; enclave[27]=0x63 shadow[28]=0x33; enclave[28]=0x33 shadow[29]=0x00; enclave[29]=0x9f shadow[30]=0xc9; enclave[30]=0xc9 shadow[31]=0x9a; enclave[31]=0x9a shadow[32]=0x66; enclave[32]=0x66 shadow[33]=0x32; enclave[33]=0x32 shadow[34]=0x0d; enclave[34]=0x0d shadow[35]=0x00; enclave[35]=0xb7 shadow[36]=0x31; enclave[36]=0x31 shadow[37]=0x58; enclave[37]=0x58 shadow[38]=0x00; enclave[38]=0xa3 shadow[39]=0x5a; enclave[39]=0x5a shadow[40]=0x25; enclave[40]=0x25 shadow[41]=0x5d; enclave[41]=0x5d shadow[42]=0x05; enclave[42]=0x05 shadow[43]=0x17; enclave[43]=0x17 shadow[44]=0x58; enclave[44]=0x58 shadow[45]=0xe9; enclave[45]=0xe9 shadow[46]=0x5e; enclave[46]=0x5e shadow[47]=0xd4; enclave[47]=0xd4 shadow[48]=0xab; enclave[48]=0xab shadow[49]=0x00; enclave[49]=0xb2 shadow[50]=0xcd; enclave[50]=0xcd shadow[51]=0xc6; enclave[51]=0xc6 shadow[52]=0x9b; enclave[52]=0x9b shadow[53]=0x00; enclave[53]=0xb4 shadow[54]=0x54; enclave[54]=0x54 shadow[55]=0x11; enclave[55]=0x11 shadow[56]=0x0e; enclave[56]=0x0e shadow[57]=0x82; enclave[57]=0x82 shadow[58]=0x74; enclave[58]=0x74 shadow[59]=0x41; enclave[59]=0x41 shadow[60]=0x21; enclave[60]=0x21 shadow[61]=0x3d; enclave[61]=0x3d ** shadow[62]=0x00; enclave[62]=0xdc shadow[63]=0x87; enclave[63]=0x87 [foreshadow.c] [FAIL] Foreshadow missed 15 bytes out of 64 :/

Every time the bytes missed is diffetent.

[jovanbulck]

This is good news, this means that the basic Foreshadow attack works :tada:

The reason that every run the byte values differ, is because the enclave every time generates a new random secret to be extracted.

As explained in the Foreshadow paper, the attack needs the victim data to be in the L1D cache, so when you read 0x00 and the PoC misses some of the bytes, this means they have been evicted from L1D. This is not too surprising given this is an unoptimized PoC demo and, without TSX, the secret might be kicked from the cache due to the OS page-fault handler activity (see also the paper for some optimizations to try to better keep secrets in the cache)

[jovanbulck]

The commit above fixes the app/memcmp bugs on my machine, so closing this now. Feel free to open a (new) issue if it doesn't work for you.