search

gramineproject / examples

Sample applications configs for Gramine
BSD 3-Clause "New" or "Revised" License
29 stars 22 forks source link

Error only for the openjdk demo: "Enclave creation IOCTL failed with Input/output error (EIO)." #100

Closed BergemannFortiss closed 2 months ago

BergemannFortiss commented 2 months ago

Description of the problem

I have set up SGX and Gramine, and tested several examples/demos like:

All of them have worked. Then, I wanted to test the Java example (https://github.com/gramineproject/examples/tree/master/openjdk), but this always fails with an IO error when creating the enclave (see below). It is definitely linked with SGX, because when executing it without SGX, it works as expected.

I will outline what I already did (/installed) and how the context looks like below, but since the other SGX demos work, it surprises me that the simple Java one does not. Nevertheless, I assume it is my fault and a setting is wrong. Therefore, I hope that you can lead me to the problem source.

Steps to reproduce

Or rather my setup (you might not be able to reproduce it):

Platform:

Operating System: Ubuntu 20.04.6 LTS
Kernel: Linux 5.15.0-105-generic
Architecture: x86-64

(Therefore, I should have the in-built SGX driver and does not need to install one myself, right?)

SGX support (SDK and PSW installed):

$ is-sgx-available 
SGX supported by CPU: true
SGX1 (ECREATE, EENTER, ...): true
SGX2 (EAUG, EACCEPT, EMODPR, ...): false
Flexible Launch Control (IA32_SGXPUBKEYHASH{0..3} MSRs): true
SGX extensions for virtualizers (EINCVIRTCHILD, EDECVIRTCHILD, ESETCONTEXT): false
Extensions for concurrent memory management (ETRACKC, ELDBC, ELDUC, ERDINFO): false
CET enclave attributes support (See Table 37-5 in the SDM): false
Key separation and sharing (KSS) support (CONFIGID, CONFIGSVN, ISVEXTPRODID, ISVFAMILYID report fields): false
Max enclave size (32-bit): 0x80000000
Max enclave size (64-bit): 0x1000000000
EPC size: 0x5d80000
SGX driver loaded: true
AESMD installed: true
SGX PSW/libsgx installed: true
#PF/#GP information in EXINFO in MISC region of SSA supported: false
#CP information in EXINFO in MISC region of SSA supported: false

Important SGX files existing (not sure if also /dev/sgx_vepc must be present?):

$ ls -ltr /dev/sgx*
crw-rw-rw- 1 root root    10, 125 Mai  3 14:27 /dev/sgx_enclave
crw-rw---- 1 root sgx_prv 10, 126 Mai  3 14:27 /dev/sgx_provision

/dev/sgx:
total 0
lrwxrwxrwx 1 root root 14 Mai  3 14:27 enclave -> ../sgx_enclave
lrwxrwxrwx 1 root root 16 Mai  3 14:27 provision -> ../sgx_provision

Installed DCAP:

$ apt list --installed | grep dcap

gramine-ratls-dcap/focal,now 1.7 amd64 [installed,automatic]
libsgx-dcap-quote-verify/unknown,now 1.21.100.3-focal1 amd64 [installed,automatic]

Gramine installed:

$ apt list gramine
Listing... Done
gramine/focal,now 1.7 amd64 [installed]

The make step for the Java example works:

$ make SGX=1 DEBUG=1
javac -d . MultiThreadMain.java
gramine-manifest \
    -Dlog_level=debug \
    -Darch_libdir=/lib/x86_64-linux-gnu \
    -Dentrypoint=/usr/lib/jvm/java-11-openjdk-amd64/bin/java \
    java.manifest.template >java.manifest
gramine-sgx-sign \
    --manifest java.manifest \
    --output java.manifest.sgx
Attributes (required for enclave measurement):
    size:        0x400000000
    edmm:        False
    max_threads: 64
SGX remote attestation:
    None
Memory:
    00000003fff26000-0000000400000000 [REG:R--] (manifest) measured
    00000003ffd26000-00000003fff26000 [REG:RW-] (ssa) measured
    00000003ffce6000-00000003ffd26000 [TCS:---] (tcs) measured
    00000003ffca6000-00000003ffce6000 [REG:RW-] (tls) measured
    00000003ffc66000-00000003ffca6000 [REG:RW-] (stack) measured
    00000003ffc26000-00000003ffc66000 [REG:RW-] (stack) measured
    00000003ffbe6000-00000003ffc26000 [REG:RW-] (stack) measured
    00000003ffba6000-00000003ffbe6000 [REG:RW-] (stack) measured
    00000003ffb66000-00000003ffba6000 [REG:RW-] (stack) measured
    00000003ffb26000-00000003ffb66000 [REG:RW-] (stack) measured
    00000003ffae6000-00000003ffb26000 [REG:RW-] (stack) measured
    00000003ffaa6000-00000003ffae6000 [REG:RW-] (stack) measured
    00000003ffa66000-00000003ffaa6000 [REG:RW-] (stack) measured
    00000003ffa26000-00000003ffa66000 [REG:RW-] (stack) measured
    00000003ff9e6000-00000003ffa26000 [REG:RW-] (stack) measured
    00000003ff9a6000-00000003ff9e6000 [REG:RW-] (stack) measured
    00000003ff966000-00000003ff9a6000 [REG:RW-] (stack) measured
    00000003ff926000-00000003ff966000 [REG:RW-] (stack) measured
    00000003ff8e6000-00000003ff926000 [REG:RW-] (stack) measured
    00000003ff8a6000-00000003ff8e6000 [REG:RW-] (stack) measured
    00000003ff866000-00000003ff8a6000 [REG:RW-] (stack) measured
    00000003ff826000-00000003ff866000 [REG:RW-] (stack) measured
    00000003ff7e6000-00000003ff826000 [REG:RW-] (stack) measured
    00000003ff7a6000-00000003ff7e6000 [REG:RW-] (stack) measured
    00000003ff766000-00000003ff7a6000 [REG:RW-] (stack) measured
    00000003ff726000-00000003ff766000 [REG:RW-] (stack) measured
    00000003ff6e6000-00000003ff726000 [REG:RW-] (stack) measured
    00000003ff6a6000-00000003ff6e6000 [REG:RW-] (stack) measured
    00000003ff666000-00000003ff6a6000 [REG:RW-] (stack) measured
    00000003ff626000-00000003ff666000 [REG:RW-] (stack) measured
    00000003ff5e6000-00000003ff626000 [REG:RW-] (stack) measured
    00000003ff5a6000-00000003ff5e6000 [REG:RW-] (stack) measured
    00000003ff566000-00000003ff5a6000 [REG:RW-] (stack) measured
    00000003ff526000-00000003ff566000 [REG:RW-] (stack) measured
    00000003ff4e6000-00000003ff526000 [REG:RW-] (stack) measured
    00000003ff4a6000-00000003ff4e6000 [REG:RW-] (stack) measured
    00000003ff466000-00000003ff4a6000 [REG:RW-] (stack) measured
    00000003ff426000-00000003ff466000 [REG:RW-] (stack) measured
    00000003ff3e6000-00000003ff426000 [REG:RW-] (stack) measured
    00000003ff3a6000-00000003ff3e6000 [REG:RW-] (stack) measured
    00000003ff366000-00000003ff3a6000 [REG:RW-] (stack) measured
    00000003ff326000-00000003ff366000 [REG:RW-] (stack) measured
    00000003ff2e6000-00000003ff326000 [REG:RW-] (stack) measured
    00000003ff2a6000-00000003ff2e6000 [REG:RW-] (stack) measured
    00000003ff266000-00000003ff2a6000 [REG:RW-] (stack) measured
    00000003ff226000-00000003ff266000 [REG:RW-] (stack) measured
    00000003ff1e6000-00000003ff226000 [REG:RW-] (stack) measured
    00000003ff1a6000-00000003ff1e6000 [REG:RW-] (stack) measured
    00000003ff166000-00000003ff1a6000 [REG:RW-] (stack) measured
    00000003ff126000-00000003ff166000 [REG:RW-] (stack) measured
    00000003ff0e6000-00000003ff126000 [REG:RW-] (stack) measured
    00000003ff0a6000-00000003ff0e6000 [REG:RW-] (stack) measured
    00000003ff066000-00000003ff0a6000 [REG:RW-] (stack) measured
    00000003ff026000-00000003ff066000 [REG:RW-] (stack) measured
    00000003fefe6000-00000003ff026000 [REG:RW-] (stack) measured
    00000003fefa6000-00000003fefe6000 [REG:RW-] (stack) measured
    00000003fef66000-00000003fefa6000 [REG:RW-] (stack) measured
    00000003fef26000-00000003fef66000 [REG:RW-] (stack) measured
    00000003feee6000-00000003fef26000 [REG:RW-] (stack) measured
    00000003feea6000-00000003feee6000 [REG:RW-] (stack) measured
    00000003fee66000-00000003feea6000 [REG:RW-] (stack) measured
    00000003fee26000-00000003fee66000 [REG:RW-] (stack) measured
    00000003fede6000-00000003fee26000 [REG:RW-] (stack) measured
    00000003feda6000-00000003fede6000 [REG:RW-] (stack) measured
    00000003fed66000-00000003feda6000 [REG:RW-] (stack) measured
    00000003fed26000-00000003fed66000 [REG:RW-] (stack) measured
    00000003fece6000-00000003fed26000 [REG:RW-] (stack) measured
    00000003feca6000-00000003fece6000 [REG:RW-] (stack) measured
    00000003fec96000-00000003feca6000 [REG:RW-] (sig_stack) measured
    00000003fec86000-00000003fec96000 [REG:RW-] (sig_stack) measured
    00000003fec76000-00000003fec86000 [REG:RW-] (sig_stack) measured
    00000003fec66000-00000003fec76000 [REG:RW-] (sig_stack) measured
    00000003fec56000-00000003fec66000 [REG:RW-] (sig_stack) measured
    00000003fec46000-00000003fec56000 [REG:RW-] (sig_stack) measured
    00000003fec36000-00000003fec46000 [REG:RW-] (sig_stack) measured
    00000003fec26000-00000003fec36000 [REG:RW-] (sig_stack) measured
    00000003fec16000-00000003fec26000 [REG:RW-] (sig_stack) measured
    00000003fec06000-00000003fec16000 [REG:RW-] (sig_stack) measured
    00000003febf6000-00000003fec06000 [REG:RW-] (sig_stack) measured
    00000003febe6000-00000003febf6000 [REG:RW-] (sig_stack) measured
    00000003febd6000-00000003febe6000 [REG:RW-] (sig_stack) measured
    00000003febc6000-00000003febd6000 [REG:RW-] (sig_stack) measured
    00000003febb6000-00000003febc6000 [REG:RW-] (sig_stack) measured
    00000003feba6000-00000003febb6000 [REG:RW-] (sig_stack) measured
    00000003feb96000-00000003feba6000 [REG:RW-] (sig_stack) measured
    00000003feb86000-00000003feb96000 [REG:RW-] (sig_stack) measured
    00000003feb76000-00000003feb86000 [REG:RW-] (sig_stack) measured
    00000003feb66000-00000003feb76000 [REG:RW-] (sig_stack) measured
    00000003feb56000-00000003feb66000 [REG:RW-] (sig_stack) measured
    00000003feb46000-00000003feb56000 [REG:RW-] (sig_stack) measured
    00000003feb36000-00000003feb46000 [REG:RW-] (sig_stack) measured
    00000003feb26000-00000003feb36000 [REG:RW-] (sig_stack) measured
    00000003feb16000-00000003feb26000 [REG:RW-] (sig_stack) measured
    00000003feb06000-00000003feb16000 [REG:RW-] (sig_stack) measured
    00000003feaf6000-00000003feb06000 [REG:RW-] (sig_stack) measured
    00000003feae6000-00000003feaf6000 [REG:RW-] (sig_stack) measured
    00000003fead6000-00000003feae6000 [REG:RW-] (sig_stack) measured
    00000003feac6000-00000003fead6000 [REG:RW-] (sig_stack) measured
    00000003feab6000-00000003feac6000 [REG:RW-] (sig_stack) measured
    00000003feaa6000-00000003feab6000 [REG:RW-] (sig_stack) measured
    00000003fea96000-00000003feaa6000 [REG:RW-] (sig_stack) measured
    00000003fea86000-00000003fea96000 [REG:RW-] (sig_stack) measured
    00000003fea76000-00000003fea86000 [REG:RW-] (sig_stack) measured
    00000003fea66000-00000003fea76000 [REG:RW-] (sig_stack) measured
    00000003fea56000-00000003fea66000 [REG:RW-] (sig_stack) measured
    00000003fea46000-00000003fea56000 [REG:RW-] (sig_stack) measured
    00000003fea36000-00000003fea46000 [REG:RW-] (sig_stack) measured
    00000003fea26000-00000003fea36000 [REG:RW-] (sig_stack) measured
    00000003fea16000-00000003fea26000 [REG:RW-] (sig_stack) measured
    00000003fea06000-00000003fea16000 [REG:RW-] (sig_stack) measured
    00000003fe9f6000-00000003fea06000 [REG:RW-] (sig_stack) measured
    00000003fe9e6000-00000003fe9f6000 [REG:RW-] (sig_stack) measured
    00000003fe9d6000-00000003fe9e6000 [REG:RW-] (sig_stack) measured
    00000003fe9c6000-00000003fe9d6000 [REG:RW-] (sig_stack) measured
    00000003fe9b6000-00000003fe9c6000 [REG:RW-] (sig_stack) measured
    00000003fe9a6000-00000003fe9b6000 [REG:RW-] (sig_stack) measured
    00000003fe996000-00000003fe9a6000 [REG:RW-] (sig_stack) measured
    00000003fe986000-00000003fe996000 [REG:RW-] (sig_stack) measured
    00000003fe976000-00000003fe986000 [REG:RW-] (sig_stack) measured
    00000003fe966000-00000003fe976000 [REG:RW-] (sig_stack) measured
    00000003fe956000-00000003fe966000 [REG:RW-] (sig_stack) measured
    00000003fe946000-00000003fe956000 [REG:RW-] (sig_stack) measured
    00000003fe936000-00000003fe946000 [REG:RW-] (sig_stack) measured
    00000003fe926000-00000003fe936000 [REG:RW-] (sig_stack) measured
    00000003fe916000-00000003fe926000 [REG:RW-] (sig_stack) measured
    00000003fe906000-00000003fe916000 [REG:RW-] (sig_stack) measured
    00000003fe8f6000-00000003fe906000 [REG:RW-] (sig_stack) measured
    00000003fe8e6000-00000003fe8f6000 [REG:RW-] (sig_stack) measured
    00000003fe8d6000-00000003fe8e6000 [REG:RW-] (sig_stack) measured
    00000003fe8c6000-00000003fe8d6000 [REG:RW-] (sig_stack) measured
    00000003fe8b6000-00000003fe8c6000 [REG:RW-] (sig_stack) measured
    00000003fe8a6000-00000003fe8b6000 [REG:RW-] (sig_stack) measured
    00000003fe84c000-00000003fe89c000 [REG:R-X] (code) measured
    00000003fe89c000-00000003fe8a6000 [REG:RW-] (data) measured
    0000000000010000-00000003fe84c000 [REG:RWX] (free)
Measurement:
    f154e5a850b8af817c843f4079b35ee45d0588b0626f7af6e592f0883ef42da7

In the error message, it is said that my manifest requires CPU features (e.g. sgx.require_avx512) that are not available on this platform. However, no special CPU features are required for this demo, right? Therefore, this should not break anything:

[sgx.cpu_features]
avx = "unspecified"
avx512 = "unspecified"
amx = "unspecified"
mpx = "disabled"
pkru = "disabled"

Expected results

Terminal output:

$ gramine-sgx java -Xmx8G MultiThreadMain
Gramine is starting. Parsing TOML manifest file, this may take some time...

Final Count is: 10000

Actual results

Terminal output:

$ gramine-sgx java -Xmx8G MultiThreadMain
Gramine is starting. Parsing TOML manifest file, this may take some time...
(host_main.c:969:load_enclave) debug: Gramine parsed TOML manifest file successfully
(host_framework.c:213:create_enclave) error: Enclave creation IOCTL failed with Input/output error (EIO). Probably your manifest requires CPU features (e.g. `sgx.require_avx512`) that are not available on this platform.
(host_main.c:349:initialize_enclave) error: Creating enclave failed: Input/output error (EIO)
(host_main.c:1271:main) error: load_enclave() failed with error: Input/output error (EIO)

Gramine Examples commit hash

https://github.com/gramineproject/examples/commit/ceba8e9da926a3cf83d821ea78fb131a5b43ea9d

mkow commented 2 months ago

In the error message, it is said that my manifest requires CPU features (e.g. sgx.require_avx512) that are not available on this platform. However, no special CPU features are required for this demo, right?

There are, see: https://github.com/gramineproject/examples/blob/ceba8e9da926a3cf83d821ea78fb131a5b43ea9d/openjdk/java.manifest.template#L27-L28

And is-sgx-available says that your CPU is missing this feature:

#CP information in EXINFO in MISC region of SSA supported: false

So, I'm closing this issue, because everything works as expected, but feel free to ask any more questions (related to this problem) if you have any.

// btw. thank you for taking time to report this in a clear and high-quality way, I appreciate :)

BergemannFortiss commented 2 months ago

@mkow Thanks for pointing this out. I missed that "use_exinfo" refers to another feature for which I have to check the CPU support.

In the docs, I found "sgx.insecureallow_memfaults_without_exinfo" (https://gramine.readthedocs.io/en/latest/manifest-syntax.html#sgx-exinfo, discussed here: https://github.com/gramineproject/gramine/pull/1744) of which a note says that it "is provided only to allow debugging/testing on old CPUs that do not support the EXINFO feature". This is exactly my situation, since I just want to quickly try out Java with SGX before getting suitable hardware. Therefore, I replaced sgx.use_exinfo with sgx.insecureallow_memfaults_without_exinfo. Now, the enclave is built and the pages are added, but after that everything freezes and the program is killed (probably a memory issue).

Is it therefore right, when I assume that sgx.insecure__allow_memfaults_without_exinfo cannot solve/cover the missing EXINFO support in case of Java applications (with the trade-of of reduced security) and you can definitely only run Java application with a correctly supported sgx.use_exinfo? Or is there another workaround to test a small Java application with a CPU that does not support EXINFO? Thanks in advance.

mkow commented 2 months ago

I missed that "use_exinfo" refers to another feature for which I have to check the CPU support.

Yeah, it doesn't have "require" in the name because of a slightly different semantics - false means that this feature won't be used at all, even if it's actually available.

This is exactly my situation, since I just want to quickly try out Java with SGX before getting suitable hardware.

Ah, then it's ok to use it, just ensure you won't accidentally ship something with this configuration to production. I actually forgot that we left that switch for debugging/development purposes, sorry.

Now, the enclave is built and the pages are added, but after that everything freezes and the program is killed (probably a memory issue).

It's probably running out of memory, Java likes to preallocate gigabytes of memory relying on lazy allocation, which isn't supported yet by Gramine. Lazy allocation requires EDMM support which your CPU doesn't have and https://github.com/gramineproject/gramine/pull/1513, which is currently blocked on bugs in the SGX Linux driver. Without that you'll need to have enough RAM+swap to store the whole preallocated buffer.

Also, remember that even if you free some RAM/add swap it will be quite slow. You only have 93 MB of EPC, which means a lot of swapping to non-SGX RAM (this isn't an issue on newer server CPUs which practically don't have a limit on EPC size).

Is it therefore right, when I assume that sgx.insecure__allow_memfaults_without_exinfo cannot solve/cover the missing EXINFO support in case of Java applications

No, as far as I remember it should work. I think you just don't have enough RAM+swap (see above).