search

tosc-rs / mnemos

An Operating System for Building Small Computers
https://mnemos.dev
Apache License 2.0
241 stars 16 forks source link

platform(d1): persistent boot #139

Closed hawkw closed 5 months ago

hawkw commented 1 year ago

Currently, we can only boot MnemOS on the D1 by writing the kernel directly into the D1's DDR using xfel. This is not ideal. Instead, we should be able to put the kernel image either in flash or on the SD card, and have the device bootload itself when it comes out of reset.

per @jamesmunns in issue #120:

This is probably going to involve a couple things:

  • Figuring out how to prepare a binary image that the D1 will recognize and start booting from
  • Writing some kind of early stage bootloader (could be the whole kernel at first lol)
  • Figuring out how much of the SD card the hardware will load for us
  • Initializing the DDR3 memory
  • Booting without the help of XFEL

I think the oreboot/rustsbi crates have at least figured out part of this (preparing an image, and initializing the DDR3 memory), though it may have been from a SPI flash and not an SD card, though the stock MQ-Pro boards do not come with SPI flash, so we would have to solder our own on.

per @orangecms in https://github.com/tosc-rs/mnemos/issues/120#issuecomment-1616595458:

All that the D1 needs to boot from any media is the special eGON header. On SD cards, it needs to be at a specific offset. See here further down: https://linux-sunxi.org/Allwinner_Nezha#U-Boot

We have the header implemented in oreboot; see the sunxi/nezha mainboard directory, ant bt0/main.rs under it. A checksum is filled in via xtask.

So, it sounds like we can boot directly from SD if we can make an eGON header and put it at the right offset. Adding SPI flash to the MnemOS breakout board is (probably?) not necessary? The sunxi wiki explains the eGON header format here: https://linux-sunxi.org/EGON

(sidenote: every time I see "eGON header" i immediately think of Egon from Ghostbusters)

hawkw commented 1 year ago

Oreboot's D1 stage0 lives here. In particular, we can see their eGON header here: https://github.com/oreboot/oreboot/blob/2ce5277b86d07b1e1c680397a38cbaa59ce43a6a/src/mainboard/sunxi/nezha/bt0/src/main.rs#L65-L80 They have a cargo xtask that does most of the flashing work: https://github.com/oreboot/oreboot/blob/main/xtask/src/sunxi/nezha.rs

hawkw commented 1 year ago

And, @luojia65 also has a D1 stage0: https://github.com/luojia65/test-d1-flash-bare/blob/bd653f4a7cee4b4a95af4ce82e43beda1de9cf38/test-d1-flash-bt0/src/main.rs (fairly similar to Oreboot). The eGON header is here: https://github.com/luojia65/test-d1-flash-bare/blob/bd653f4a7cee4b4a95af4ce82e43beda1de9cf38/test-d1-flash-bt0/src/main.rs#L76-L91

The eGON header's checksum and length fields are populated by a cargo xtask similar to Oreboot's, the code for calculating this is here: https://github.com/luojia65/test-d1-flash-bare/blob/bd653f4a7cee4b4a95af4ce82e43beda1de9cf38/xtask/src/main.rs#L155-L193

hawkw commented 1 year ago

@luojia65's repo looks like it burns the boot code into flash. Oreboot's xtask looks like it just outputs a .bin file with the payload. If we want to boot directly from SD with no SPI flash (assuming that's possible?), we would probably need a slightly different workflow for preparing an image that can be written to a SD card.

hawkw commented 1 year ago

As a side note, there's a bunch of fairly nice general-purpose code for cargo xtasks in inoculate (mycelium's stupid build tool), which we could either just copy-paste or turn into a lib crate in the mycelium repo we can depend on...

orangecms commented 1 year ago

Correct. The mask ROM of the D1 - like with most Allwinner SoCs - goes around and probes different storage media, including SD card. If all fail to present the eGON header, it falls back to FEL mode. For the SD card, the header needs to start at an offset of 16*8k (0x2_0000), or at a later one (please see the sunxi wiki for ref).

For history: We worked with luojia to get the whole thing up and running, eventually brought it into oreboot. Note that oreboot is still being developed further. If you want DRAM (512MB works with the current code), and then load a payload from an SD card (I started that but it's unfinished, see the PRs and feel free to help out), then using oreboot without SBI would be a perfect choice for MnemOS.

Loading from flash is implemented already and works. In fact, I had already demoed booting MnemOS from flash a year back: https://twitter.com/OrangeCMS/status/1553080059526889472

hawkw commented 1 year ago

That's great, thanks for all the additional context @orangecms! I think we would love to potentially contribute back to upstream Oreboot.

jspngh commented 1 year ago

Here is minimal bootloader based on the oreboot nezha boot0. You can put it on a SD card and it will load 1MB of data (the mnemos binary) into DRAM. oreboot-bt0.tar.gz without UART output oreboot-bt0.tar.gz with UART output

To create the SD card

sudo dd if=oreboot-nezha-bt0.bin of=/dev/sdX bs=1024 seek=8
sudo dd if=mnemos.bin of=/dev/sdX bs=1024 seek=40

I tested it on the lichee rv (dock pro), but hopefully it will also work on the MangoPi mq pro. I can verify that mnemos is running, but there are still some issue that I need to tackle.

For instance, if I make this change in bin/lichee-rv.rs

     let mut p = unsafe { d1_pac::Peripherals::steal() };
-    let uart = unsafe { kernel_uart(&mut p.CCU, &mut p.GPIO, p.UART0) };
+    let mut uart = unsafe { kernel_uart(&mut p.CCU, &mut p.GPIO, p.UART0) };
     let spim = unsafe { kernel_spim1(p.SPI_DBI, &mut p.CCU, &mut p.GPIO) };
     let i2c0 = unsafe { twi::I2c0::lichee_rv_dock(p.TWI2, &mut p.CCU, &mut p.GPIO) };
     let timers = Timers::new(p.TIMER);
     let dmac = Dmac::new(p.DMAC, &mut p.CCU);
     let plic = Plic::new(p.PLIC);

+    uart.write(b"Hello World?");
+
     let d1 = D1::initialize(timers, uart, spim, dmac, plic, i2c0).unwrap();

I can see Hello World? being printed, but nothing else, while I do see 'hello' messages when loading with xfel. So the UART seems to be initialized properly, maybe the issue is with the DMA.

I'll dig into this, seems like a good opportunity to get familiar with the code.

hawkw commented 1 year ago

I've also tested booting from SD on the MangoPi MQ Pro, using @jspngh's instructions.

I can confirm that the UART doesn't seem to work, and SPI also doesn't seem to work (the Beepy's SHARP display never comes up). However, we do successfully cycle the Beepy's RGB LED colors, so I2C does seem to work. This further implicates some kind of issue with DMA transfers, since both the UART and SPI drivers use the DMAC, but the I2C driver (currently) does not.

jspngh commented 1 year ago

With the merge of #192, it is now possible to boot from an SD card using the oreboot bt0. However this currently is hardcoded to a Linux payload, so for now we will have to use a slightly modified version which you can find in the mnemos branch of my oreboot fork.

I want to propose both a short-term and long-term plan for persistent boot:

@hawkw @jamesmunns please let me know what you think about these ideas.

jamesmunns commented 1 year ago

So, in terms of licensing:

Can we add a single module / file which has a GPL license instead of the default MIT + Apache? Another approach could be to develop our own reverse-engineered version based on the assembly file from Allwinner (but I think this file was technically also GPL licensed?).

Not generally, no. The way GPL generally works is that if you "ship it together", like in a binary image, then everything that gets shipped together is GPL.

However, we could have a GPL bootloader project, that uses both the GPL DRAM code as well as the MIT+Apache2 mnemos components. I'd prefer that this lives in a separate repo in the tosc-rs space, and not in the main mnemos/ repo.

Just as a question, does the xboot dram init also come from the GPL'd unlicensed chunks? Or is there enough there to build our own dram init code?

jamesmunns commented 1 year ago

If MnemOS were to become a real RISC-V OS :TM:, it will need to use the supervisor-binary interface.

I'm not against this, but I admit to not really understanding what SBI gets us. I think it's a sort of abstraction layer that would allow us to potentially run mnemos on more targets, but AFAIU it doesn't generally cover hardware drivers for all of those targets? Or if it does, we would need to "fall back" to using probably blocking system call style drivers instead of native async drivers?

jspngh commented 1 year ago

So, in terms of licensing:

Can we add a single module / file which has a GPL license instead of the default MIT + Apache? Another approach could be to develop our own reverse-engineered version based on the assembly file from Allwinner (but I think this file was technically also GPL licensed?).

Not generally, no. The way GPL generally works is that if you "ship it together", like in a binary image, then everything that gets shipped together is GPL.

However, we could have a GPL bootloader project, that uses both the GPL DRAM code as well as the MIT+Apache2 mnemos components. I'd prefer that this lives in a separate repo in the tosc-rs space, and not in the main mnemos/ repo.

Just as a question, does the xboot dram init also come from the GPL'd unlicensed chunks? Or is there enough there to build our own dram init code?

As far as I know, everyone started from the same assembly file that is also included in xboot. The sunxi wiki lists this as "with a GPL2 license tag from the very beginning", but I would have to go look for the original source to confirm this. I don't think anyone would take issue if we create another Rust version based on this assembly file and license it MIT + Apache2, since xboot is currently also shipping it under an MIT license. Creating a separate repo could also work.

If MnemOS were to become a real RISC-V OS :TM:, it will need to use the supervisor-binary interface.

I'm not against this, but I admit to not really understanding what SBI gets us. I think it's a sort of abstraction layer that would allow us to potentially run mnemos on more targets, but AFAIU it doesn't generally cover hardware drivers for all of those targets? Or if it does, we would need to "fall back" to using probably blocking system call style drivers instead of native async drivers?

My limited understanding: the supervisor binary is given access to the peripheral memory region, so it can implement its own (async) drivers. The SBI is only used for the limited set of functionality that is listed on the page I linked. It's something we can think about, but it'll be a long time before any decision has to be made. If MnemOS were to really take off, this is a request that could pop up again if somebody wants/needs it.

orangecms commented 1 year ago

Hey there, let me chime in again. :-)

The SBI is just a soft platform definition matter. You can bring your own full stack and part of that would be your own handlers for calling from S-mode into M-mode (which is what the SBI spec tries to standardize).

Running an OS in M-mode is possible just like that and something we deliberately support in oreboot. Our approach is to either let you fully bring your M-mode payload, so you would build oreboot with your own payload as a build argument to it, or leverage LinuxBoot, which means we use a Linux kernel as the payload that comes with a little userland that implements boot loaders. For that, we use https://u-root.org - so that you can put your OS on an SD card, on the network, on a USB drive or whatever you like, and the u-root enivronment can load any of that because it uses the drivers that Linux already has for all those peripherals. Does that make sense?

Addendum: Linux expects an SBI, so we implement one. It is an optional feature in oreboot and being kept minimal. I.e., when you bring your M-mode payload, you build oreboot without passing --supervisor.

jspngh commented 1 year ago

Thanks for the explanation @orangecms. I imagined the SBI was more fundamental when it came to operating systems on RISC-V, but it seems we are free to choose what we want :smile:.

hawkw commented 5 months ago

we can boot from a SD card now, so i think this is potentially close-able?

jspngh commented 5 months ago

Indeed, maybe I should have done that as part of PR #190? I can close it now, or you can go ahead and do it too of course.