The goal of this issue is to run Linux on top of Mirage, using a virtualized OpenSBI.
Context
OpenSBI is a standard RISC-V firmware, that is a software that runs in M-mode (the highest privilege mode) and provides a standardized execution environment for S-mode software (e.g. the OS). By default running a kernel on RISC-V QEMU will first load OpenSBI as the firmware, similar to how x86_64 QEMU loads SeaBIOS by default.
The goal of Mirage is to virtualize the firmware to enforce strong isolation guarantees. To do so Mirage deprivileges the firmware and virtualize privileged instructions using a trap & emulate approach. See the architecture diagram in the Mirage doc for reference.
So far we managed to run a virtualized OpenSBI with a toy kernel (which you can reproduce with just run opensbi, see readme for detailed instructions). The goal of this issue is to run a real kernel, i.e. Linux, on top of our virtualized OpenSBI.
How to proceed
[x] First get your environment ready. Follow the readme to install the required tools (Rust, just, the other tools with just install-toolchain, and a RISC-V-ready QEMU qemu-system-riscv64).
Then run Mirage with the default test firmware: just run. You should get some logs including [Info | mirage::virt] > Hello from default firmware!
[x] At this point Mirage is up and running, the next step is to run OpenSBI. You can easily run a pre-build OpenSBI with just run opensbi, which should display the OpenSBI banner and Test payload running. The pre-built binaries come form this repo that we maintain, and are used for testing purpose.
In order to run Linux we will need a custom build of OpenSBI. To do so start by downloading the OpenSBI sources. Then you will need a RISC-V C toolchain, OpenSBI support multiple of them, so pick one that is available for your OS (such as riscv64-linux-gnu-gcc most likely).
Once you have a proper toolchain it is time to start building OpenSBI. There are instructions in the OpenSBI repository, but as a summary make PLATFORM=generic CROSS_COMPILE=riscv64-linux-gnu-, where CROSS_COMPILE must be set to the name of your toolchain minus the final gcc (because riscv64-linux-gnu-gcc comes with other tools such as the linker riscv64-linux-gnu-ld).
The build will produce several binaries in build/platform/generic/firmware/, in particular fw_payload.bin which is the one used by the just run opensbi commands. The other binaries (fw_jump.bin and fw_dynamic.bin) are essentially the same, but they load the payload (the OS kernel) differently:
payload package the kernel directly inside the OpenSBI binary.
jump takes an address to jump to, it assumes the kernel was already loaded at this address.
dynamic uses some mechanisms to figure out what to do depending on the current state of the machine.
Now that we have built a fresh OpenSBI image we can run it on Mirage with just run <PATH_TO_OPENSBI>, where the path points to build/platform/generic/firmware/fw_payload.bin. You should see the OpenSBI banner and this time QEMU will not exit (because the default OpenSBI test kernel loops indefinitely), you can exit with Ctrl-A x.
If something goes wrong, such as getting a panic from Mirage when running OpenSBI, feel free to ask. Chances are this is because your toolchain does not support PIE (position independant executables), in which case you need to patch the OpenSBI configuration to change the OpenSBI start address (like in this patch).
[x] Now that you can build and run OpenSBI it is time to move on to Linux. We don't have precise instruction for this part because we never tried it so far, so it is uncharted territory. Here are some ideas on how to proceed, though:
[x] First get a working RISC-V Linux kernel. There are hopefully tutorials online, you should be able to get Linux to run using qemu-system-riscv64, the same QEMU we used for Mirage.
The user-space don't matter much, it's OK if Linux stops because it can't find a disk (if we get as far when running on Mirage that's a huge success! The hard part is going to be the kernel initialization). So you can skip the part of the tutorial that builds a disk or busybox for instance.
[x] Once you have a kernel that boots in qemu-system-riscv64 the next step is to bundle it with our custom build of OpenSBI. There are some instructions on how to do so in the OpenSBI doc (don't forget the CROSS_COMPILE argument, they didn't include it). This will produce a new OpenSBI image with Linux included.
[x] The next step is to try running that new image on Mirage, with just run <PATH_TO_OPENSBI>. We don't expect things to work right out of the box here, most likely Mirage will panic because of some un-supported feature. You can try to run OpenSBI on QEMU directly (without Mirage, the OpenSBI doc has some instructions too) to confirm that it works as expected.
[x] We are at the point where we can run an OpenSBI image that will load Linux on top of Mirage, but most likely the Linux initialization fails. The next steps are to understand what is going wrong (missing features in Mirage, bugs, something else?) and decide on a strategy to resolve those issues.
CharlyCst
commented
3 months ago
The goal of this issue is to run Linux on top of Mirage, using a virtualized OpenSBI.
Context
OpenSBI is a standard RISC-V firmware, that is a software that runs in M-mode (the highest privilege mode) and provides a standardized execution environment for S-mode software (e.g. the OS). By default running a kernel on RISC-V QEMU will first load OpenSBI as the firmware, similar to how x86_64 QEMU loads SeaBIOS by default.
The goal of Mirage is to virtualize the firmware to enforce strong isolation guarantees. To do so Mirage deprivileges the firmware and virtualize privileged instructions using a trap & emulate approach. See the architecture diagram in the Mirage doc for reference.
So far we managed to run a virtualized OpenSBI with a toy kernel (which you can reproduce with
just run opensbi, see readme for detailed instructions). The goal of this issue is to run a real kernel, i.e. Linux, on top of our virtualized OpenSBI.How to proceed
just, the other tools withjust install-toolchain, and a RISC-V-ready QEMUqemu-system-riscv64). Then run Mirage with the default test firmware:just run. You should get some logs including[Info | mirage::virt] > Hello from default firmware!just run opensbi, which should display the OpenSBI banner andTest payload running. The pre-built binaries come form this repo that we maintain, and are used for testing purpose. In order to run Linux we will need a custom build of OpenSBI. To do so start by downloading the OpenSBI sources. Then you will need a RISC-V C toolchain, OpenSBI support multiple of them, so pick one that is available for your OS (such asriscv64-linux-gnu-gccmost likely). Once you have a proper toolchain it is time to start building OpenSBI. There are instructions in the OpenSBI repository, but as a summarymake PLATFORM=generic CROSS_COMPILE=riscv64-linux-gnu-, whereCROSS_COMPILEmust be set to the name of your toolchain minus the finalgcc(becauseriscv64-linux-gnu-gcccomes with other tools such as the linkerriscv64-linux-gnu-ld). The build will produce several binaries inbuild/platform/generic/firmware/, in particularfw_payload.binwhich is the one used by thejust run opensbicommands. The other binaries (fw_jump.binandfw_dynamic.bin) are essentially the same, but they load the payload (the OS kernel) differently:payloadpackage the kernel directly inside the OpenSBI binary.jumptakes an address to jump to, it assumes the kernel was already loaded at this address.dynamicuses some mechanisms to figure out what to do depending on the current state of the machine. Now that we have built a fresh OpenSBI image we can run it on Mirage withjust run <PATH_TO_OPENSBI>, where the path points tobuild/platform/generic/firmware/fw_payload.bin. You should see the OpenSBI banner and this time QEMU will not exit (because the default OpenSBI test kernel loops indefinitely), you can exit withCtrl-A x. If something goes wrong, such as getting a panic from Mirage when running OpenSBI, feel free to ask. Chances are this is because your toolchain does not support PIE (position independant executables), in which case you need to patch the OpenSBI configuration to change the OpenSBI start address (like in this patch).qemu-system-riscv64, the same QEMU we used for Mirage. The user-space don't matter much, it's OK if Linux stops because it can't find a disk (if we get as far when running on Mirage that's a huge success! The hard part is going to be the kernel initialization). So you can skip the part of the tutorial that builds a disk or busybox for instance.qemu-system-riscv64the next step is to bundle it with our custom build of OpenSBI. There are some instructions on how to do so in the OpenSBI doc (don't forget the CROSS_COMPILE argument, they didn't include it). This will produce a new OpenSBI image with Linux included.just run <PATH_TO_OPENSBI>. We don't expect things to work right out of the box here, most likely Mirage will panic because of some un-supported feature. You can try to run OpenSBI on QEMU directly (without Mirage, the OpenSBI doc has some instructions too) to confirm that it works as expected.