search

keystone-enclave / keystone

Keystone Enclave (QEMU + HiFive Unleashed)
Other
465 stars 134 forks source link

keystone_enclave: failed to allocate 12328 page(s) #259

Open andreaskuster opened 3 years ago

andreaskuster commented 3 years ago

Congrats on this great project!

I am currently in the process of deploying Keystone to the CVA6 RISC-V CPU.

The bootup process and kernel module loading process works:

Welcome to Buildroot
buildroot login: root
Password:
login[110]: root login on 'console'
# ls
hello-native.ke     keystone-driver.ko
hello.ke            tests.ke
# insmod keystone-driver.ko
[   76.418447] keystone_driver: loading out-of-tree module taints kernel.
[   76.622934] keystone_enclave: keystone enclave v1.0.0

The hello-native example as well,

# ./hello-native.ke
Verifying archive integrity... All good.
Uncompressing Keystone Enclave Package
Enclave said: "Hello World"

however, the hello example does not print any output (apart from package extraction).

# ./hello.ke
Verifying archive integrity... All good.
Uncompressing Keystone Enclave Package

Furthermore, the real trouble comes with the tests example

# ./tests.ke
Verifying archive integrity... All good.
Uncompressing Keystone Enclave Package
testing stack
[  977.651272] ------------[ cut here ]------------
[  977.658860] WARNING: CPU: 0 PID: 317 at mm/page_alloc.c:4794 __alloc_pages_nodemask+0x16c/0x198
[  977.670755] Modules linked in: keystone_driver(O)
[  977.677916] CPU: 0 PID: 317 Comm: test-runner Tainted: G           O      5.7.0-dirty #7
[  977.688829] epc: ffffffe0006e1342 ra : ffffffe000655e42 sp : ffffffe0000b5a10
[  977.698236]  gp : ffffffe00146b400 tp : ffffffe00009df00 t0 : 0000003fffa0b768
[  977.708577]  t1 : 0000003fffa0b7f0 t2 : 0000003fffa0b9c8 s0 : ffffffe0000b5a70
[  977.718467]  s1 : 0000000003028000 a0 : 0000000000000000 a1 : 000000000000000e
[  977.728778]  a2 : 0000000000000000 a3 : 0000000000000000 a4 : 0000000000000000
[  977.738665]  a5 : 000000000000000a a6 : 0000000000062ee0 a7 : 000000000000001d
[  977.748559]  s2 : 00000000ffffffff s3 : 0000000003028000 s4 : ffffffe000080c00
[  977.758409]  s5 : 0000000000000cc4 s6 : 0000000000000000 s7 : ffffffe00002e3f0
[  977.768739]  s8 : 00000000000d32f0 s9 : 00000000000e6c70 s10: 00000000000f55d0
[  977.778626]  s11: 0000000000000000 t3 : 000000000009c680 t4 : 0000000000000005
[  977.788472]  t5 : 0000000000000006 t6 : 0000000000040000
[  977.796532] status: 0000000200000120 badaddr: 0000000000009002 cause: 0000000000000003
[  977.807128] ---[ end trace 89c35b8f1e59c2a2 ]---
[  977.813794] keystone_enclave: failed to allocate 12328 page(s)
[  977.821700] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  977.938064] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-870812472 != 12345)
testing loop
[  978.317284] keystone_enclave: failed to allocate 12328 page(s)
[  978.325553] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  978.362871] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-849136440 != 54321)
testing malloc
[  978.739401] keystone_enclave: failed to allocate 12329 page(s)
[  978.748492] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  978.779364] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-940858168 != 11411)
testing long-nop
[  979.153849] keystone_enclave: failed to allocate 12332 page(s)
[  979.162138] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  979.198496] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-577030968 != 12345)
testing fibonacci
[  979.574274] keystone_enclave: failed to allocate 12328 page(s)
[  979.582592] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  979.614380] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-323017528 != 14930352)
testing fib-bench
[  979.991672] keystone_enclave: failed to allocate 12328 page(s)
[  980.000864] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  980.031557] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
testing attestation
[  980.407800] keystone_enclave: failed to allocate 12329 page(s)
[  980.416982] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  980.454307] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-454429496 != 0)
testing untrusted
[  980.829912] keystone_enclave: failed to allocate 12329 page(s)
[  980.838180] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  980.867745] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-157428536 != 13)
testing data-sealing
[  981.249438] keystone_enclave: failed to allocate 12329 page(s)
[  981.257693] keystone_enclave: failed to initialize epm
ioctl error: Cannot allocate memory
[  981.294403] keystone_enclave: invalid enclave id
[Keystone SDK] /home/andreas/Desktop/keystone/sdk/src/host/Enclave.cpp:433 : failed to run enclave - ioctl() failed
[FAIL] enclave returned a wrong value (-1124096824 != 0)
#

Do you have any hints or ideas on why this happens?

dayeol commented 3 years ago

Hello,

12328 pages are equal to about 50 MB of memory. I don't have much information about your machine, but if your Linux cannot allocate this much contiguous memory, it will fail to create an enclave.

Your Linux may have fail to allocate the CMA memory region. Please take a closer look at Linux boot message, it should say something like the following if it was successful (XXX K cma-reserved):

[    0.000000] Memory: 931620K/2095104K available (6486K kernel code, 4184K rwdata, 4096K rodata, 235K init, 318K bss, 114908K reserved, 1048576K cma-reserved)

If it says 0 K cma-reserved, it means that Linux fail to allocate CMA for some reason.

Please let me know how much of CMA-reserved memory your Linux has.

Quick fix for this issue is to reduce the size of enclave to be less than 2MB, because if there's no CMA then Keystone driver will just use buddy allocator (which cannot allocate over 2MB).

andreaskuster commented 3 years ago

Yes, you were exaclty right. Porting from the SiFive HiFive Unleashed dev board, the CMA value was set to CONFIG_CMA_SIZE_MBYTES=1024, which was too much for my Xilinx Genesys 2 dev board/FPGA. Reducing it to 256MiB successfully mitigated this issue. Thanks!

# insmod keystone-driver.ko
[   69.725113] keystone_driver: loading out-of-tree module taints kernel.
[   69.928969] keystone_enclave: keystone enclave v1.0.0
# ./tests.ke
Verifying archive integrity... All good.
Uncompressing Keystone Enclave Package
testing stack
testing loop
testing malloc
testing long-nop
testing fibonacci
testing fib-bench
testing attestation
Attestation report is invalid
testing untrusted
Enclave said: hello world!
Enclave said: 2nd hello world!
Enclave said value: 13
Enclave said value: 20
testing data-sealing
Enclave said: Sealing key derivation successful!

The only remaining example that does not succeed yet is the attestation example (see above).

I did find your thread discussing the same problem (but for firesim), but couldn't find an straight-forward solution so far.

Could you give me some pointer on where to start/look at? Alternatively I saw that you are extending the attestation example in the master sdk branch. Is this fundamentally based on the same low-level primitives, or would it make sense to get this running instead of the 'old' one in the test which only checks the signature, see: https://github.com/keystone-enclave/keystone-sdk/blob/24a5ed369ac0606aac486382a0855a459d9362ed/examples/tests/test-runner.cpp#L49 ?

jarkkojs commented 1 year ago

I get a different sort of issue with both 256MB and 512MB CMA:

# ./tests.ke
Verifying archive integrity... All good.
Uncompressing Keystone Enclave Package
testing stack

After this nothing happens but I can still exit the program. Not even 20 minutes after startup.

Any ideas what could go wrong?