pRuntime: CPU hanging up with any ecall from rocket

[h4x3rotab]

Reproduce

1. Compile the latest pruntime (6e904bc4f587fb4a870eb9ff88fae1c878604f9f) in SGX=HW mode
2. Start ./app under pruntime/bin
3. Run ./pruntime/scripts/console get_info, and pruntime hangs up

Reproduced on J4015 CPU. Some with FLC support can be walked around by switching to DCAP driver, but some cannot.

SGX driver

• DCAP driver works fine
• Linux SGX driver has the hanging up problem
  • Not all IAS driver has the problem. We confirmed NUC8i5beh can work with IAS driver.

Known checks

• Running with sgx-gdb shows SIGBUS on any ocall (e.g. println!) or memory allocation.
• Upgrading PSW doesn't help

• A binary search on git tree reveals the suspicious commits:

  595e843c9b9436ce4ec0b75ff4866a7a5c428274 pruntime: seal private keys <-- first skipped bb82651734252c31ddf8278467547d34c633a49b follow latest substrate <-- skipped 0e159ef491b1b426be06081f5e12d90ba0ca9501 pruntime: make it work again <-- first bad

[PHA-SYSOPS]

This happens too when a 'console init' is issued , sending the payload:

curl -sgX POST -H Content-Type: application/json http://127.0.0.1:8000/init_runtime -d {"input":{"skip_ra": false, "bridge_genesis_info_b64": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEDmnA2xKydrb2HsTrcVvPiQyA5LvV0LPHa2dh9q6ktgDFwoudZe3t+PYTAU5HROaYrFX54eG2MCC8p3PTBETFAAIiNw0F9UFjsS0UD4MEuoaCom+IA/piSJCPUM0AU+msO4BAAAAAAAAANF8LXgj6/Jg/ROPLX4n0RTAFF2Wi1/1AGEl8kFPra5pAQAAAAAAAAAMoQKALhCA33I0FGiDoLZ6HBWl1uCIt+sLgUbPlfMJqUk/gukhEt6AviHkl5KFGndUmA+ClBT2kPSvmBOvZTWowWjNYfynHU6AkLJ6ULmAAZPol0tbWaCiuvxqTk9Sx+9cqgrVL94lrrmAUJegqkIzp2A6LPkZouRRsKgiY4Wu92V8JXrn3aSXrw2AXDYZ0c8CICTMvasQ+rEpErmfEmg+BzH19s/zJX4LP8ZtAYKmqACACKEICOLbrasjYGHJaC9qmpHtUgEnM5k2qVJpNxn9mbdQXHh0cmluc2ljX2luZGV4EAAAAACAc0yvcsUiYcma5kSPZKxrMxbyDufisOfMmIsX1bDxfHedAWRyYW5kcGFfYXV0aG9yaXRpZXNJAQEIiNw0F9UFjsS0UD4MEuoaCom+IA/piSJCPUM0AU+msO4BAAAAAAAAANF8LXgj6/Jg/ROPLX4n0RTAFF2Wi1/1AGEl8kFPra5pAQAAAAAAAAA="}, "nonce": {"id": 3931}}

[PHA-SYSOPS]

strace:

[pid 4035] accept4(3, {sa_family=AF_INET, sin_port=htons(52460), sin_addr=inet_addr("172.17.0.1")}, [128->16], SOCK_CLOEXEC) = 6 [pid 4035] setsockopt(6, SOL_SOCKET, SO_RCVTIMEO, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0 [pid 4035] setsockopt(6, SOL_SOCKET, SO_SNDTIMEO, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0 [pid 4035] getpeername(6, {sa_family=AF_INET, sin_port=htons(52460), sin_addr=inet_addr("172.17.0.1")}, [128->16]) = 0 [pid 4035] fcntl(6, F_DUPFD_CLOEXEC, 0) = 7 [pid 4035] fcntl(7, F_GETFD) = 0x1 (flags FD_CLOEXEC) [pid 4035] recvfrom(7, "POST /init_runtime HTTP/1.1\r\nHos"..., 4096, 0, NULL, NULL) = 958 [pid 4035] setsockopt(7, SOL_SOCKET, SO_RCVTIMEO, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0 [pid 4035] fcntl(7, F_DUPFD_CLOEXEC, 0) = 8 [pid 4035] fcntl(8, F_GETFD) = 0x1 (flags FD_CLOEXEC) [pid 4035] setsockopt(8, SOL_SOCKET, SO_RCVTIMEO, "\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0 [pid 4035] write(1, "\33[34m\33[32mPOST\33[0m\33[34m \33[34m/in"..., 88) = 88 [pid 4035] write(1, " \33[1;49;39m=>\33[0m \33[34mMatche"..., 174) = 174 [pid 4035] close(8) = 0 [pid 4035] write(1, "{\n \"input\": {\n \"bridge_genes"..., 840) = 840 [pid 4035] write(1, "}\n", 2) = 2 [pid 4035] mmap(NULL, 4198400, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff06c52d000 [pid 4035] gettid() = 26 [pid 4035] --- SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7ff05480c000} --- [pid 4035] rt_sigprocmask(SIG_SETMASK, [BUS], [], 8) = 0 [pid 4035] rt_sigaction(SIGBUS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7ff06ecdd8a0}, NULL, 8) = 0 [pid 4035] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 4035] rt_sigreturn({mask=[]}) = 3

[h4x3rotab]

Closing for now because it's not common any more