CCF nodes cannot start in SGX mode

[thempp66]

Describe the bug CCF nodes cannot start in release mode. Below is the output:

FileNotFoundError: [Errno 2] No such file or directory: '/project/evm4ccf/repo/cloak-tee/build/workspace/sandbox_common/0.pem'

2022-07-20T06:52:17.974429Z        0   [trace] ../src/enclave/enclave.h:155         | Creating RPC actors / ffi
2022-07-20T06:52:17.974434Z        0   [trace] F/include/ccf/endpoint_registry.h:69 | Parsed a templated endpoint: /acme-challenge/{token} became /acme-challenge/([^/]+)
2022-07-20T06:52:17.974440Z        0   [trace] F/include/ccf/endpoint_registry.h:72 | Component names are: token
2022-07-20T06:52:17.974444Z        0   [trace] ../src/enclave/enclave.h:174         | Initialize node
2022-07-20T06:52:17.974448Z        0   [debug] ../CCF/src/ds/state_machine.h:44     | [NodeState] Advancing to state 1 (from 0)
2022-07-20T06:52:17.974453Z        0   [info ] ./CCF/src/enclave/rpc_sessions.h:223 | Setting max open sessions on interface "primary_rpc_interface" (127.0.0.1:8000) to [1000, 1010] and endorsement authority to Service
2022-07-20T06:52:17.974459Z        0   [trace] ../src/enclave/enclave.h:218         | Creating node with start_type Start
2022-07-20T06:52:17.974463Z        0   [info ] ../CCF/src/node/node_state.h:1862    | Node TLS connections now accepted
2022-07-20T06:52:17.974466Z        0   [trace] F/include/ccf/endpoint_registry.h:69 | Parsed a templated endpoint: /network/nodes/{node_id} became /network/nodes/([^/]+)
2022-07-20T06:52:17.974470Z        0   [trace] F/include/ccf/endpoint_registry.h:72 | Component names are: node_id

Azure Quote Provider: libdcap_quoteprov.so [ERROR]: Could not retrieve environment variable for 'AZDCAP_DEBUG_LOG_LEVEL'
2022-07-20T06:54:31+0000.157201Z [(H)ERROR] tid(0x7fcac827f200) | quote3_error_t=SGX_QL_NO_PLATFORM_CERT_DATA
 (oe_result_t=OE_PLATFORM_ERROR) [/source/openenclave/host/sgx/sgxquote.c:oe_sgx_qe_get_target_info:706]
2022-07-20T06:54:31+0000.157213Z [(H)ERROR] tid(0x7fcac827f200) | :OE_PLATFORM_ERROR [/source/openenclave/host/sgx/quote.c:sgx_get_qetarget_info:37]
2022-07-20T06:54:31.243979Z        0   [fail ] ../src/enclave/enclave.h:222         | Error starting node: Failed to get evidence: OE_PLATFORM_ERROR
2022-07-20T06:54:31.244010Z        0   [trace] ../src/enclave/enclave.h:187         | Finishing RDRAND engine
2022-07-20T06:54:31.244015Z        0   [trace] ../src/enclave/enclave.h:191         | Shutting down enclave
2022-07-20T06:54:31.244041Z        100 [fail ] ../src/host/main.cpp:563             | An error occurred when creating CCF node: InternalError

It seems that the problem is similar to #3747

Expected behavior Node started sucessfully.

Environment information Version of the code being used, versions of dependencies, relevant environment info where applicable (OS, SGX driver...). commit b21c981044d4674750d6cdd16dbf76c6f78bcce6 ccf-3.0.0-dev1 Linux 0ed7cf5f0771 5.10.60-9.al8.x86_64 #1 SMP Mon Sep 6 21:30:37 CST 2021 x86_64 x86_64 x86_64 GNU/Linux

LINUX KERNEL WITH BUILT-IN SGX SUPPORT (5.11+):
5.10.60-9.al8.x86_64

AESM DAEMON:
Not running

PSW INFO:
libsgx-ae-id-enclave/unknown,now 1.14.100.3-focal1 amd64 [installed,automatic]
libsgx-ae-pce/unknown,now 2.17.100.3-focal1 amd64 [installed]
libsgx-ae-qe3/unknown,now 1.14.100.3-focal1 amd64 [installed]
libsgx-ae-qve/unknown,now 1.14.100.3-focal1 amd64 [installed]
libsgx-dcap-ql-dev/unknown,now 1.14.100.3-focal1 amd64 [installed]
libsgx-dcap-ql/unknown,now 1.14.100.3-focal1 amd64 [installed]
libsgx-enclave-common/unknown,now 2.17.100.3-focal1 amd64 [installed]
libsgx-headers/unknown,now 2.17.100.3-focal1 amd64 [installed,automatic]
libsgx-pce-logic/unknown,now 1.14.100.3-focal1 amd64 [installed]
libsgx-qe3-logic/unknown,now 1.14.100.3-focal1 amd64 [installed]
libsgx-quote-ex/unknown,now 2.17.100.3-focal1 amd64 [installed]
libsgx-urts/unknown,now 2.17.100.3-focal1 amd64 [installed]

DCAP CLIENT INFO:
az-dcap-client/now 1.11.2 amd64 [installed,upgradable to: 1.11.2]

SGX INFO:
CPU supports SGX_FLC:Flexible Launch Control
CPU supports Software Guard Extensions:SGX2
SGX1
MaxEnclaveSize_64: 2^(56)
CPU supports Key Sharing & Separation (KSS): true
EPC size on the platform: 33285996544

Additional context

Please tell me if you need more environment informations. Thanks for your help!

[achamayou]

Hi @thempp66, this is an issue with https://github.com/openenclave/openenclave/, please check your configuration and open an issue there if this still happens.

[thempp66]

Thanks for your reply. Compare to CCF CI pipline, I notice that the difference is my linux kernal version is 5.10 and CCF CI pipline's is higher than 5.11. Could this be the casue of this issue? @achamayou