Crash when running performance test.

[andre-w-fischer]

Overview

A default (but unverified) installation of the 1 GB EdglessDB docker image crashes when processing requests from the mysqlslap tool. With the 4 GB image, the test finishes successfully.

Setup

Start EdgelessDB as outlined on https://docs.edgeless.systems/edgelessdb/getting-started/quickstart-sgx. I used an unverified setup. I.e.

docker run -t --name my-edb -p3306:3306 -p8080:8080 --device /dev/sgx_enclave --device /dev/sgx_provision ghcr.io/edgelesssys/edgelessdb-sgx-1gb

Run performance test like this

    mysqlslap --host=127.0.0.1 --port=3306 --user=root --ssl-cert cert.pem --ssl-key key.pem        \
              --auto-generate-sql --concurrency=50                                                  \
              --number-of-queries=150000                                                            \
              --number-char-cols=10 --number-char-cols=10 -vv --auto-generate-sql-add-autoincrement \
              --auto-generate-sql-write-number=200000

Result

The mysqlslap client prints

Building Create Statements for Auto
Building Query Statements for Auto
Parsing engines to use.
mysqlslap: Error when connecting to server: Lost connection to MySQL server at 'reading initial communication packet', system error: 0

The output of the server is

debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libsgx-dcap-default-qpl.
(Reading database ... 4914 files and directories currently installed.)
Preparing to unpack .../libsgx-dcap-default-qpl_1.15.100.3-focal1_amd64.deb ...
Unpacking libsgx-dcap-default-qpl (1.15.100.3-focal1) ...
Setting up libsgx-dcap-default-qpl (1.15.100.3-focal1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
PCCS_URL: https://172.17.0.1:8081/sgx/certification/v3/
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2023/01/19 07:54:21 EdgelessDB v0.3.2 (619e16871072635985030083b3fc3536c2d7df01)
[EDB] 2023/01/19 07:54:21 DB has not been initialized, waiting for manifest.
ERROR: dcap_quoteprov: [ERROR]: [QPL] Failed to get quote config. Error code is 0xb010

[get_platform_quote_cert_data ../qe_logic.cpp:378] Error returned from the p_sgx_get_quote_config API. 0xe047
ERROR: quote3_error_t=SGX_QL_PLATFORM_UNKNOWN
 (oe_result_t=OE_PLATFORM_ERROR) [openenclave-src/host/sgx/sgxquote.c:oe_sgx_qe_get_target_info:706]
[EDB] 2023/01/19 07:54:21 Failed to get quote: OE_PLATFORM_ERROR
[EDB] 2023/01/19 07:54:21 Attestation will not be available.
[EDB] 2023/01/19 07:54:21 HTTP REST API listening on :8080
2023/01/19 07:55:50 http: TLS handshake error from 172.17.0.1:44612: EOF
[EDB] 2023/01/19 07:56:32 initializing ...
2023-01-19  7:56:32 0 [Note] edb (server 10.6.11-MariaDB) starting as process 37 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2023/01/19 07:56:39 EdgelessDB v0.3.2 (619e16871072635985030083b3fc3536c2d7df01)
[EDB] 2023/01/19 07:56:39 starting up ...
2023-01-19  7:56:39 0 [Note] edb (server 10.6.11-MariaDB) starting as process 37 ...
ERROR: dcap_quoteprov: [ERROR]: [QPL] Failed to get quote config. Error code is 0xb010

[get_platform_quote_cert_data ../qe_logic.cpp:378] Error returned from the p_sgx_get_quote_config API. 0xe047
ERROR: quote3_error_t=SGX_QL_PLATFORM_UNKNOWN
 (oe_result_t=OE_PLATFORM_ERROR) [openenclave-src/host/sgx/sgxquote.c:oe_sgx_qe_get_target_info:706]
ERROR: :OE_ENCLAVE_ABORTING [openenclave-src/host/calls.c:_call_enclave_function_impl:56]
./edb: line 3:    37 Aborted                 (core dumped) erthost "$DIR/edb-enclave.signed" "$@"
: command not found
: command not found

Notes

• When I use the ghcr.io/edgelesssys/edgelessdb-sgx-4gb image, the load test finishes successfully.
• A similar test (with larger values for the query counts) produces an error when running against a vanilla MariaDB that runs under gramine and kubernetes. The server side error message (from gramine) claims that the number of threads (configured to be 250) is exceeded but MariaDB should use only 50 (per configuration). Might be a problem with the EPC memory (which is configured to be 4 GiB). Something similar could be the root cause for the crash in EdgelessDB.

System information

OS: Ubuntu 22.04.1 LTS x86_64 Kernel: 5.15.0-56-generic CPU: Intel Xeon Platinum 8352Y (128) @ 3.400GHz Memory: 7012MiB / 128581MiB

[thomasten]

Hi, this is an out of memory error and therefore expected with the 1 GB image. It's not easy to see this, unfortunately. You can enable debug logging if you want to make it visible.

[andre-w-fischer]

Is it possible to configure EdgelessDB in a way, that it does not try to allocate more memory than is available in the enclave?

[thomasten]

This is currently not possible and I guess it would require nontrivial changes in the underyling MariaDB/RocksDB forks. For example, the original DBs can use temporary disk storage.