LekKit / RVVM

The RISC-V Virtual Machine
GNU General Public License v3.0
871 stars 61 forks source link

Instruction how to run Linux #57

Closed X547 closed 5 months ago

X547 commented 1 year ago

Are there any recommended Linux disk images and firmware sets that are confirmed to work? I tried your kernels from this comment, but after switching to NVMe it do not boot anymore.

LekKit commented 1 year ago

Are there any recommended Linux disk images and firmware sets that are confirmed to work? I tried your kernels from this comment, but after switching to NVMe it do not boot anymore.

As about firmware, the generic OpenSBI works fine. You may download prebuilt binaries from their Releases repo section. The fw_jump.bin bootrom is used to boot into the SBI payload (S-mode kernel like Linux or U-Boot).

After that, you have a bunch of choices for both kernel and userland

Here is my personal Linux build config for RV64: linux_6.0_rv64_config.txt Place it into a kernel source directory as .config and build: make ARCH=riscv INSTALL_MOD_STRIP=1 CROSS_COMPILE=riscv64-linux-gnu- -j`nproc` Image

Here is a built kernel Image from that config: linux_6.0.zip

Be aware this build incorporates out-of-tree fixes which apparently no one in the mailing list wants to merge... See my rant here: https://lore.kernel.org/all/76741e1c-056d-d3ae-fc20-eeb8f6e996f6@gmail.com/, etc, but you probably shouldn't care about that

Btw, there is a nice OpenSBI fork called OpenSBI-H which introduces Hypervisor extension support to platforms that don't have any hardware support for it. This allows running KVM inside RVVM guests, for example, but this is really experimental and I'm afraid not going to upstream OpenSBI.

If you have any other questions, I'll gladly help.

LekKit commented 1 year ago

My usual approach to booting a distro with my custom kernel:

This is of course really simplified, dumb, does not reflect the boot flow of real RISC-V boards, but that's easy to get working and see in action. As for more proficient users, they may use SBI->U-Boot->GRUB->Linux - whatever, I'm not very good at explaining that, but it sure will work as well.

LekKit commented 1 year ago

Can I close this issue? Should I add more info to README perhaps, since that already covers some basics on starting up machines? I don't think distributing binary kernels compiled for my personal use and tailored to current RVVM spec is a good idea, and I would like to hear any suggestions.

X547 commented 1 year ago

The best would be an instruction how to run Linux in RVVM with using only released binaries and without building/patching anything (except building RVVM itself). Including distribution names and download URLs.

Non-Linux (Windows, MacOS, Haiku etc.) users may also want to run RVVM and it is hard to build Linux there.

LekKit commented 1 year ago

The best would be an instruction how to run Linux in RVVM with using only released binaries and without building/patching anything (except building RVVM itself). Including distribution names and download URLs.

Yeah, but that's complicated with RISC-V for now. The first stage, OpenSBI is pretty straightforward (Generic builds from their releases work pretty much on any RV machine) Unfortunately there is no such thing as "generic RISC-V U-Boot", and the "qemu board" build which is used in a lot of places doesn't even support NVMe (I don't know who did this, but it's extremely stupid. It's the only thing lacking for this build to work in RVVM, and also breaks when emulating NVMe in QEMU.) Most Linux distributions either don't provide an installation medium (Only having a repo and a complicated way of bootstrapping) or have different tailored images for Unmatched/QEMU, etc. My best bet is Ubuntu at this point. Then there are FreeBSD/OpenBSD which have installation images and work fine in RVVM, but have terrible set of prebuilt drivers (They don't even support simple-framebuffer), which leaves you with a guest running over UART with no gfx/networking. I still have no luck running Haiku from nightly builds over U-Boot, and this feels like RVVM bug 😢. It should work since the same set of devices (ACLINT/PLIC/NS16550A/PCI/NVMe) & same FDT works in QEMU (Alright, what i mean is the kernel doesn't panic). I will investigate this further because having upstream Haiku support would be beneficial, and RVVM should not be inferior to QEMU at least when it comes to the common devices they have.

X547 commented 1 year ago

With Haiku I try to achieve possibility to run the same Haiku disk image on all supported RISC-V platforms, both hardware and virtual. For hardware platforms UEFI/FDT is used (usually u-boot based and shipped with hardware). For virtual machines (TinyEMU, QEmu, RVVM) Haiku boot loader complied as firmware (haiku_loader.riscv, no SBI) can be used. Currently haiku_loader.riscv is broken for RVVM because of missing NVMe support (only VirtIO block and ATA are currently supported, maybe I will implement NVMe support this week).

Despite that haiku_loader.riscv is used as a firmware, it is very portable because of convention to load and execute firmware at 0x80000000 and common entry point signature void _start(uintptr_t hart, void *fdt).

Linux have strange tradition to ship image with FDT data and u-boot that makes it non-portable. So I promote to use UEFI and FDT stored in hardware.

LekKit commented 1 year ago

So far there are 3 ways for proper & transparent support of all distros, firmwares, etc:

Linux have strange tradition to ship image with FDT data and u-boot that makes it non-portable. So I promote to use UEFI and FDT stored in hardware.

Not really, the FDT my VM generates is passed all over through SBI, U-Boot and to the guest kernel. It's simply the fact that most images are tailored to specific target(s) or aren't tested well.

LekKit commented 1 year ago

Currently haiku_loader.riscv is broken for RVVM because of missing NVMe support (only VirtIO block and ATA are currently supported, maybe I will implement NVMe support this week).

ATA still works in RVVM but not available from CLI. You can patch main.c to attach ATA devices instead of NVMe, or you can tell me and I will add some kind of -ata argument specifically for that. I just don't think ATA on RISC-V is a thing at all, or is it?

X547 commented 1 year ago

ATA still works in RVVM but not available from CLI. You can patch main.c to attach ATA devices instead of NVMe, or you can tell me and I will add some kind of -ata argument specifically for that.

At my previous work on RVVM support I adapted ATA support for boot loader and kernel so it can pass up to GUI. But I got some strange problems with timer interrupts so thread scheduler freeze.

I just don't think ATA on RISC-V is a thing at all, or is it?

In theory you can use PATA PCIe card on Hifive Unmatched for example. But it is rare case.

LekKit commented 1 year ago

At my previous work on RVVM support I adapted ATA support for boot loader and kernel so it can pass up to GUI. But I got some strange problems with timer interrupts so thread scheduler freeze.

Ahh. Was this 0.4? It did not pass wfi tests back then, there were some related commits (c7cda30, 896a68a) that fixed it.

LekKit commented 1 year ago

In theory you can use PATA PCIe card on Hifive Unmatched for example. But it is rare case.

Welp, U-Boot doesn't even have working drivers for that then. They are using some x86 port IO specific stuff, do not support ATA over PCI and do not enumerate ATA PIO from FDT. Never got it to work.

LekKit commented 1 year ago

See https://lekkit.github.io/test/index.html for some nightmare fuel. Jokes aside, I consider making some kind of a demo like that, for people who visit RVVM repo and want to quickly look what it's like.

adamierymenko commented 1 year ago

I'd like to test this. Is there just an archive somewhere that I could just boot from, replicating the example in the README? This would be a great thing to link. Or did I miss it? I'm figuring it out but it'd be awesome if you could download a tarball just to test drive this thing.

igor-tsiglyar commented 1 year ago

@LekKit Hi, I've been trying to run the ubuntu disk I have installed using qemu and this instruction.

I have run it with qemu using

qemu-system-riscv64 -machine virt -m 4G -smp cpus=4 -nographic \
    -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.bin \
    -kernel /usr/lib/u-boot/qemu-riscv64_smode/u-boot.bin \
    -netdev user,id=net0,hostfwd=tcp::2222-:22 \
    -device virtio-net-device,netdev=net0 \
    -drive file=disk.img,format=raw,if=virtio \
    -device virtio-rng-pci

I am trying to run it on this vm using

./RVVM/release.linux.x86_64/rvvm_x86_64 /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.bin \
    -m 4G -smp 4 -nogui \
    -kernel /usr/lib/u-boot/qemu-riscv64_smode/u-boot.bin \
    -image disk.img

I fails on u boot stage with

Device 0: unknown device
scanning bus for devices...

Device 0: unknown device
No ethernet found.
No ethernet found.

I have noticed there is no networking support yet, but perhaps there is a way to make u boot ignore this and run regardless?

Thank you!

LekKit commented 1 year ago

I have noticed there is no networking support yet, but perhaps there is a way to make u boot ignore this and run regardless?

This is a known issue on U-Boot side (More precisely, the config for QEMU board). They build support for VirtIO block device found in QEMU, but not for NVMe drives (Even though QEMU may emulate NVMe as well). I will soon provide version of U-Boot for RVVM to sort things out, for now you may use this (Or build U-Boot yourself): u-boot_s.bin.zip

I also recommend raising JIT cache size with -jitcache 64M for perf reasons

I have noticed there is no networking support yet

There is, but QEMU U-Boot again has no drivers for it, and doesn't need to know about it's existence at all

igor-tsiglyar commented 1 year ago

@LekKit That one worked, thanks again and special thanks for this promising project - the performance seems to be better that qemu's indeed

LekKit commented 1 year ago

I'd like to test this. Is there just an archive somewhere that I could just boot from, replicating the example in the README? This would be a great thing to link. Or did I miss it? I'm figuring it out but it'd be awesome if you could download a tarball just to test drive this thing.

This was already explained here: https://github.com/LekKit/RVVM/issues/57#issuecomment-1320433138

TL;DR: Use upstream OpenSBI from their releases, U-Boot build provided in this thread, then download some upstream distro like Ubuntu. Other option incurs using prebuilt kernel from the same thread, then preparing a rootfs with your distro of choise (Explained in https://github.com/LekKit/RVVM/issues/57#issuecomment-1320444106). This eliminates the need for complex U-Boot->EFI->Distro kernel boot flow, instead the kernel is directly ran after the firmware.

As for the general guide & prebuilt images: I'm gonna do this in time with v0.5 release, this is gonna take a bunch of weeks probably. Thanks for patience)

LekKit commented 1 year ago

@LekKit That one worked, thanks again and special thanks for this promising project - the performance seems to be better that qemu's indeed

I'm glad you like it)

adamierymenko commented 1 year ago

I got it going, and the performance seems very good. What are the plans for networking support in this? I have some really interesting use case ideas.

LekKit commented 1 year ago

I got it going, and the performance seems very good. What are the plans for networking support in this? I have some really interesting use case ideas.

Networking separates into NIC device support (Guest sees it & passes ethernet packets through it) and TAP interface (Host-side thing for virtual ethernet to interact with outside world)

What's working now:

How to use userspace networking (WIP):

For Linux TAP (Beware: Complicated):

adamierymenko commented 1 year ago

For user space networking it'd be good to have some way to use this as a library too, to embed into other applications.

Why not do virtio devices inside the VM? Support for those is in most OSes and AFAIK they are very well suited for this kind of thing being designed to interface between a guest and a hypervisor.

LekKit commented 1 year ago

For user space networking it'd be good to have some way to use this as a library too, to embed into other applications.

Do you mean the TAP interface? See src/devices/tap_api.h, it is probably trivial to expose this as a public API. Not really complicated either.

Why not do virtio devices inside the VM? Support for those is in most OSes and AFAIK they are very well suited for this kind of thing being designed to interface between a guest and a hypervisor.

See https://github.com/LekKit/RVVM/issues/55#issuecomment-1321074438 for my (subjective) rant about virtio. Main points:

LekKit commented 1 year ago

@X547 A note about Linux guests:

The reason for slow GL is softpipe. Drawing a few quads is insanely slower than writing pixel data directly, to the point where desktop renders at ~1 FPS. That bites us in many places still... Most DEs (KDE, Gnome, Xfce) use GL in one place or another, making them unusable unless you disable it (If that's even possible). Hope for llvmpipe to be faster if it's ported at some point, I'll consider actively helping with that (A MR is already open in Mesa repo). It'd be also great to support native GPU acceleration in RVVM, but no idea how to do that (yet).

image

Salakar commented 1 year ago

See lekkit.github.io/test/index.html for some nightmare fuel.

@LekKit Is WASM also something you plan to support longterm for RVVM - I know the performance is worse off in wasm but for my use case, it's actually fast enough (in fact much faster than other projects like this that target WASM). Happy to sponsor you too 😄 (please let me know how)

Also are you able to share the flags you used to compile the demo?

LekKit commented 1 year ago

See lekkit.github.io/test/index.html for some nightmare fuel.

@LekKit Is WASM also something you plan to support longterm for RVVM

Yes, the Emscripten target reuses 90% of other POSIXy code. RVVM only uses a tiny bit of platform-dependent code abstracted in a nice way, therefore I do not plan on any breakage.

There are some unintentional issues with this target, however (See #71), like perf and overall usability, but I hope they would be figured out at some point. I do have higher priorities than improving WASM target. but it's not going to get worse, that's it.

There are also plans for more advanced interpreter approaches which could speed up WASM, non-JIT native targets and JIT tracing phase.

It should be even possible to implement rvjit_wasm backend, but only if someone is drunk enough to actually pull this off (And maintain it!)

Sad thing is: Different WASM runtimes have different perf and I am not in control of that. On Firefox, RVVM Web Demo boots in ~1.5 secs for me, whereas on Chromium it is 6 seconds. Doom demo is unplayable on Chromium for me 3:

LekKit commented 1 year ago

Also are you able to share the flags you used to compile the demo?

  1. Get a working firmware, kernel, root image
  2. Get a pre.js file which contains arguments to rvvm-cli; And a shell file which is basically an HTML page that embeds it (optional)
  3. make CC=emcc LDFLAGS="--pre-js pre.js --preload-file fw_jump.bin --preload-file linux_6.2 --preload-file rootfs.img --shell-file shell_minimal.html"
  4. Use coi-serviceworker & HTTPS to actually get pthreads to work in modern paranoidal browsers (RVVM might work single-threaded but it's awful)

I've updated the demo to latest RVVM commit and uploaded all the required stuff to rebuild it See LekKit/test@97be0dd, LekKit/test@adb46ef

LekKit commented 1 year ago

Happy to sponsor you too smile (please let me know how)

Github disallows to receive payments in my region. I'm not too sad about it though.

If you like this project and want to help, there are things you can do ^^,

Salakar commented 1 year ago

@LekKit Thanks for sharing the build scripts for the demo, super helpful. My plan would be to use RVVM without a gui, in terminal/shell only mode through something like xterm.js (though I'm still to figure this out) - since all the things I'd need to emulate are shell based only. (I've noticed a recent commit disables this for emscripten)

Other things I still have to figure out for my use case:

I'm fairly new to C & wasm so it's going to take me a while, sorry in advance for any noob questions 😅

Github disallows to receive payments in my region. I'm not too sad about it though.

That's a shame 😞 I can also sponsor through crypto also, saw you mentioned it somewhere 🙃 let me know how

LekKit commented 1 year ago

I've noticed a recent commit disables shell for emscripten

Emscripten enables POSIX code paths, and RVVM is usually hooked to an actual terminal on those systems. Default Emscripten console however is very limited and slows RVVM significantly, so there was nothing useful disabled - just explicitly disabling something that was broken to begin with.

In terminal/shell only mode

I am currently working on some kind of UART/VT API which would either get you serial IO or screen characters from an internal VT. This would bring proper VT support on older Windows or... you get it, Emscripten (And other cool usecases)

  • exposing emscripten FS to the VM so I can read and write to it in the guest and in JS land

This is somewhat tough since the VM accesses raw block devices holding their own FS and stuff, same as your usual hard drive or whatever. You might want to use networking mounts or investigate 9p and how to bring support for it, if you actually need file interoperability.

  • networking support in emscripten (have seen some other projects like this do it via a proxy)

Yea it is definitely possibe. There are 2 ways: Getting tap_user socket-based backend to work under WASM, or writing some kind of tap_proxy which passes raw Ethernet packets to the proxy and outer world.

See src/networking.c for internal RVVM socket lib (Abstracts shenanigans like different types, code and workarounds for different hosts). It should be noted Emscripter doesn't advertise proper select() syscall support, which means no native socket multiplexing... Emulating it through busy-polling is possible, but very inefficient

As for writing a separate tap_proxy - it is less work on RVVM side, but requires a separate proxy specifically written for it which would act as a WebSocket<->Ethernet relay. I'm not experienced on WebSocket stuff so far.

LekKit commented 1 year ago

Question: Where should preferred SBI / U-Boot / Linux builds be hosted? In releases, CI artifacts or just a separate link?

I've published a U-Boot patch to support RVVM board & NVMe boot properly: LekKit/patches-misc@ef2bb2b There is also a tiny kernel config (Supposed to be used as SBI payload): LekKit/patches-misc@fc64fb8

@fish4terrisa-MSDSM any suggestions?

fish4terrisa-MSDSM commented 1 year ago

Build in ci ,and release one artifacts in releases is a good idea,the ci can be config as build once commited

fish4terrisa-MSDSM commented 1 year ago

Build in ci ,and release one artifacts in releases is a good idea,the ci can be config as build once commited

I recommend circleci.

hizzlekizzle commented 11 months ago

On a related note, I was interested in checking out the libretro integration but wasn't sure how to go from a compiled core lib to actually booting something. Any hints?

LekKit commented 11 months ago

On a related note, I was interested in checking out the libretro integration but wasn't sure how to go from a compiled core lib to actually booting something. Any hints?

hizzlekizzle commented 11 months ago

Awesome. Thanks for the tips!

fewletter commented 8 months ago

Hi, I am a newcomer of the project. I am trying to build RVVM as riscv32, and I have a problem of running it. I have used buildroot to generate the root filesystem image rootfs.ext2 with .config.zip and busybox.config.zip, get fw_jump.bin from the OpenSBI release and create the linux kernel linux_rv32 from the linux. However, when I run the command below.

RVVM/release.linux.riscv$ ./rvvm_riscv fw_jump.bin -kernel ../linux_rv32 -rv32 -image ../rootfs.ext2
...
[    0.297044] NET: Registered PF_PACKET protocol family
[    0.302607] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.306567] ALSA device list:
[    0.307740]   No soundcards found.
[    0.310877] VFS: Cannot open root device "nvme0n1" or unknown-block(0,0): error -6
[    0.313292] Please append a correct "root=" boot option; here are the available partitions:
[    0.315810] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.318519] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.1-rvvm #1
[    0.319944] Hardware name: RVVM v0.6-16e4574 (DT)
[    0.321245] Call Trace:
[    0.322520] [<c00032c6>] dump_backtrace+0x1a/0x22
[    0.323994] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

It seems like it couldn't find the root filesystem to mount. I don't what is happening. Is the problem cause by my root filesystem or other things and how I can build RVVM successfully?

LekKit commented 8 months ago

Hi, I am a newcomer of the project. I am trying to build RVVM as riscv32, and I have a problem of running it. I have used buildroot to generate the root filesystem image rootfs.ext2 with .config.zip and busybox.config.zip, get fw_jump.bin from the OpenSBI release and create the linux kernel linux_rv32 from the linux. However, when I run the command below.

RVVM/release.linux.riscv$ ./rvvm_riscv fw_jump.bin -kernel ../linux_rv32 -rv32 -image ../rootfs.ext2
...
[    0.297044] NET: Registered PF_PACKET protocol family
[    0.302607] debug_vm_pgtable: [debug_vm_pgtable         ]: Validating architecture page table helpers
[    0.306567] ALSA device list:
[    0.307740]   No soundcards found.
[    0.310877] VFS: Cannot open root device "nvme0n1" or unknown-block(0,0): error -6
[    0.313292] Please append a correct "root=" boot option; here are the available partitions:
[    0.315810] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.318519] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.1-rvvm #1
[    0.319944] Hardware name: RVVM v0.6-16e4574 (DT)
[    0.321245] Call Trace:
[    0.322520] [<c00032c6>] dump_backtrace+0x1a/0x22
[    0.323994] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

It seems like it couldn't find the root filesystem to mount. I don't what is happening. Is the problem cause by my root filesystem or other things and how I can build RVVM successfully?

Do you have NVMe driver enabled in the kernel? RVVM emulates NVMe for drives and it seems kernel can't find it.

This is a kernel config which includes everything needed to function on RVVM: https://github.com/LekKit/patches-misc/blob/master/linux-rvvm-configs/config_tiny_6.2 (Remember to switch to RV32 build)

fewletter commented 8 months ago

Thanks for the help. RVVM works fine now. 2023-12-30 16-58-28 screenshot

ChinYikMing commented 7 months ago
  1. Get a pre.js file which contains arguments to rvvm-cli;

I would like to give a sample for passing arguments in pre.js:

Module['arguments'] = [];
Module['arguments'].push('fw_jump.bin');
Module['arguments'].push('-k');
Module['arguments'].push('linux_6.1');
Module['arguments'].push('-i');
Module['arguments'].push('rootfs.ext2');

The prebuilt kernel and rootfs images are from here

LekKit commented 5 months ago

The new v0.6 release includes EFI U-Boot firmware and a prebuilt Arch RISC-V image.

There is also fw_jump.bin + tiny static kernel for custom buildroot builds or the likes.

Closing this for now. Reopen if any other questions arise, or preferably open a new issue.