Running Keystone on gem5

[despotx]

Describe the bug Not a bug

Question Description Hello, I would like to apply Keystone to gem5. However, gem5 only accepts two parameters: a bootable kernel file (I specified the file fw_payload.elf) and a root file system file. It can boot and run, but when executing the example "/usr/share/keystone/examples/hello.ke", it reports a PMP access error and then the system crashes. I tried using "make run," which works. This command passes three files to QEMU: bootrom.bin, fw_jump.elf, and Image. I'm not sure about the relationship between these three files and fw_payload.elf, or whether the error is caused by gem5 or these files. I hope someone can explain the relationship between these files or provide relevant documentation. Thank you.

[grg-haas]

Hi @despotx ! The rough relationship between these files is as follows:

1. The BootROM implements a mock root-of-trust in order to make attestation tests work. This includes a basic measurement of the SM, which is the next stage of the boot.
2. fw_jump.elf is a patched version of OpenSBI (a low-level RISCV monitor firmware) with support for the Keystone enclave system. This is the part that actually implements the majority of how enclaves work by utilizing PMP registers.
3. Image is the Linux kernel, pretty much unpatched for our purposes (we do have a custom, out-of-tree kernel driver).

QEMU does also use rootfs.ext2, which is a standard Buildroot-based root filesystem holding userspace applications, libraries, etc. This is eventually mounted by Image.

As to the PMP error you're getting in gem5, this could be due to a variety of reasons. I assume you're building for our generic (QEMU-based) platform? Often there need to be slight tweaks to the SM to make it work for a new platform (or simulator). Unfortunately I don't know anything about gem5's RISCV support so I can't give advice on how to fix this, but if you post the specific error you're getting I could help figure out what's happening at least.

[despotx]

@grg-haas Yeah, the platform I'm using is generic (QEMU-based). When configuring buildroot-configure, I selected the option "Include Linux as OpenSBI Payload," which generated an additional fw_payload.elf/bin after the build was completed. I specified fw_payload.elf and rootfs.ext2 for gem5, and it can successfully boot into the operating system. During startup, it displays that the sm has been initialized, and I can run regular programs once inside the system. However, when running the example "/usr/share/keystone/examples/hello.ke," I receive a warning from gem5 saying "PMP access fault." I would like to know if there is any missing data in fw_payload.elf and how I can supplement this data. Thank you.

[grg-haas]

Understood, that setup makes sense to me. A PMP access fault could happen for a number of reasons. I'll assume that this happens right after context switching into the enclave (i.e. leaving context_switch_to_enclave in sm/src/enclave.c) -- I'd love confirmation if this is in fact the case. Assuming though that this is an otherwise unmodified generic build from the current master, the main thing that I would check is that the number of PMP registers specified in the SM matches the number of such registers that gem5 implements. By default I believe we configure this to be 8 -- the specific value is PMP_N_REG in sm/src/pmp.h.

Some information that would help me debug this:

1. What's the faulting address causing the PMP access fault?
2. Do you successfully manage to context switch to the enclave runtime? Using GDB, you can check this by setting a breakpoint on context_switch_to_enclave, stepping to sbi_trap_exit, once you hit the mret instruction there do a b *$mepc, and then stepi a few times.

[despotx]

Thank you very much for your response. I investigated further, and the issue seems to be with gem5. gem5 works fine and successfully runs the sample program using fw_payload.elf and rootfs.ext2. However, there might be some issues with certain functionalities provided by gem5, which are causing the PMP access exceptions. I also verified it, and Keystone can still execute properly when the number of PMP registers specified by "sm" is fewer than what the hardware provides.