UEFI?

[johalun]

Hi! Thanks for this great blog series. It's really awesome and helping me a lot in learning how an OS works.

Do you have any plans of adding support for UEFI boot? I assume the assembly would be much simpler since UEFI setup alot of that for you, including preparing a graphical framebuffer to draw on.

As for the UEFI boot loader, I assume grub-efi can be used to load a kernel from a fat/ext/ufs partition. Or, perhaps the kernel can be put on the ESP and loaded directly by the UEFI BIOS.

If I find the time, maybe I'll play around with it and do a PR. Just wanted to make sure we're not doing double work..

[IsaacWoods]

Instead, it provides a EFI_SIMPLE_TEXT_OUPUT_PROTOCOL that can be used to print text to the screen. Unfortunately, this protocol is only available until ExitBootServices is called, so we can't use it in our kernel. (@IsaacWoods Am I understanding this correctly?)

This is correct. Unless the firmware decides otherwise, protocols are not backed by runtime-allocated memory and so can disappear at any point after you call ExitBootServices. They specifically designed EFI_GRAPHICS_OUTPUT_PROTOCOL (GOP) with our use case in mind (not having real graphics drivers) and it's designed to supply enough information that you can write to the physical framebuffer after UEFI's involvement stops - GOP is only needed to set the framebuffer up, and not to write to it, whereas EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL is involved in the actual output as well.

[64]

the legacy PIC is not supported with UEFI, so we need to migrate the tutorial to the APIC

I haven’t touched OSDev for a while, but my next plan was to flesh out the APIC crate, which I own (the crate is almost completely empty). I would be happy to give people maintainer access to the crate if it helps.

[rpjohnst]

Another option for text output, which is supported by both UEFI and BIOS, is to write to a serial port. I've used this quite a bit in the past to avoid setting up a full pixel-based framebuffer text system. It takes very little code to set up, and QEMU can forward it directly to the host console.

[phil-opp]

@IsaacWoods Thanks a lot for the explanation!

[phil-opp]

@64 That would be great, thanks a lot! Let me know if I can do anything to help.

[phil-opp]

@rpjohnst Yes, the serial port is a good option too. I personally like screen output more because it really feels like you're creating a completely new system, while serial output feels a bit like running a normal userspace program. Also, most computers today don't have an actual serial port anymore, so serial output is not accessible when running on real hardware.

[toothbrush7777777]

@rpjohnst Yes, the serial port is a good option too. I personally like screen output more because it really feels like you're creating a completely new system, while serial output feels a bit like running a normal userspace program. Also, most computers today don't have an actual serial port anymore, so serial output is not accessible when running on real hardware.

Well, not quite. Some modern PCs do still come with serial ports. A lot of ASRock's motherboards come with internal serial port connectors. I agree that writing to a frame-buffer does seem like creating a new system much more than writing to a serial port, though I found serial ports to be quite useful for debugging even with other VMMs like VirtualBox and VMware.

[phil-opp]

Interesting, I didn't know that. Still, there are many PCs and laptops without a serial port today. I fully agree that they're more useful than a framebuffer for debugging. Maybe the best approach would be to introduce both approaches (framebuffer and serial) at the beginning.

[ethindp]

I don't know if this helps, but there's an MSR that allows you to get the APIC memory address without reading ACPI tables. According to Volumes 3 and 4 of the Intel SDMs, this MSR is 1Bh. Section 10.4.4 notes that bit 11 is the enable APIC bit (or as they call it, the "APIC Global Enable" flag). The manual notes that "This flag is available in the Pentium 4, Intel Xeon, and P6 family processors. It is not guaranteed to be available or available at the same location in future Intel 64 or IA-32 processors." Additionally, it specifies that the default address (bits 35:12) is FEE00000H. Is this a reliable method of acquiring the APIC address?

[toothbrush7777777]

I don't know if this helps, but there's an MSR that allows you to get the APIC memory address without reading ACPI tables. According to Volumes 3 and 4 of the Intel SDMs, this MSR is 1Bh. Section 10.4.4 notes that bit 11 is the enable APIC bit (or as they call it, the "APIC Global Enable" flag). The manual notes that "This flag is available in the Pentium 4, Intel Xeon, and P6 family processors. It is not guaranteed to be available or available at the same location in future Intel 64 or IA-32 processors." Additionally, it specifies that the default address (bits 35:12) is FEE00000H. Is this a reliable method of acquiring the APIC address?

For those processors, probably. But I guess that would still depend on the firmware.

[ghost]

@ethindp MSRs generally should be avoided when possible for portability purposes, unless they've been around for a long time and are pretty much universal for relevant hardware (e.g. that one MSR that can be used for enabling syscall support).

@phil-opp I would suggest emulating UEFI on BIOS machines and also using UEFI on machines that support it, to drastically increase portability. Simulating UEFI in the BIOS can be done with some bootloaders people made IIRC, and Rust can interface with them if they exist and if it's done the right way because it all becomes machine code at some point.

When simulating UEFI in the BIOS, make sure that the version of UEFI that the bootloader simulates is the same as the version of UEFI being used for UEFI machines. If the versions are different, this would cause unexpected behavior.

As to making edits to the tutorials, I would suggest a third edition, as the differences between BIOS and UEFI are simply too great.

It's also worth noting that the BIOS is not going to be an exposable interface forever. Intel said they would remove it from future processors starting this year in favor of UEFI, though I'm not sure if they changed their plans since they said it.

[phil-opp]

@LiamTheProgrammer Emulating UEFI on BIOS machines sounds like a good solution, provided that this is not too much work. Otherwise, we could try to create some sort of common interface that abstracts over the BIOS/UEFI details.

I'm not really keen to create a third edition since this would be a lot of work for me and churn for readers. Instead, I think it's probably better to rewrite the individual posts that are affected by the change (the post about printing to screen and setting up hardware interrupts). We would probably need to make some adjustments to the other posts (e.g. updating the QEMU screenshots), but otherwise they should stay valid.

[ghost]

@phil-opp Creating a common interface would be extremely hard. I think BIOS emulation shouldn't be too hard, but I could be wrong. It'd still be easier than developing a common interface, though, because for it to be fully-featured it would probably have to manually implement tons of useful UEFI functions from a UEFI specification (there are many of those) for the BIOS, which would eliminate the point anyway.

And yeah, you're right about the third edition thing. I shouldn't have been inconsiderate. :/

[IsaacWoods]

@LiamTheProgrammer I'm not sure creating a common interface at a high level would be too challenging - I'm imagining things like load_file, create_memory_map and stuff being called from a common layer, which would then be implemented by both the UEFI and BIOS layers independently (along with an entry point).

It definitely sounds far far easier than emulating BIOS on UEFI without platform support (which is a large project itself, and not useful for our purposes). I'm not sure it's even possible as a normal UEFI application, as UEFI has control over the IDT and mode, so how would you go about allowing real-mode-style accesses and the BIOS software interrupts? There are also quirks that robust BIOS bootloaders have to work around, like some BIOSs not being able to load segments over 64k boundaries and such, that would add pointless overhead when we can just rely on UEFI's much better file protocols.

[phil-opp]

I just pushed the first prototype of the uefi bootloader here: https://github.com/rust-osdev/bootloader/tree/uefi

Use cargo uefi-build for building and cargo uefi-run for starting it in QEMU (requires OVMF). Like the BIOS bootloader, you need to set the path to your kernel binary in the KERNEL environment variable and the path to the Cargo.toml of your kernel in the KERNEL_MANIFEST_PATH environment variable when building.

The implementation currently only sets up a new page tables for mapping the kernel ELF file and then passes control to the kernel. No boot information is passed yet and no additional mappings (e.g. framebuffer, physical memory) are done yet.

[toothbrush7777777]

@phil-opp What is cargo uefi-build?

[phil-opp]

The commands are cargo build-uefi/cargo run-uefi, sorry. Both are just aliases defined in the .cargo/config.toml so that you don't have to remember all required cargo arguments.

[toothbrush7777777]

@phil-opp OK, that makes sense now.

[RMuskovets]

Any news?

[RMuskovets]

@IsaacWoods looks like Clover bootloader does the opposite: emulating UEFI on BIOS systems

[phil-opp]

Still working on it when I have time. Today I started implementing a new boot information struct including the memory map.

I also thought about redesigning the interface between the bootloader and bootimage crates. My basic idea is to move as much work as possible to the bootloader, so that bootimage can become a library that can be used from a local custom runner executable. The advantage of this would be that you don't have to install bootimage anymore. One requirement for this would be that cargo gains a way to reset a configured default target for a subdirectory. I opened a PR for this at https://github.com/rust-lang/cargo/pull/8638, let's see what the cargo maintainers think about it.

[ethindp]

@phil-opp That would be pretty neat. What do you need help with regarding the bootloader right now? I'm not ultra-well-versed in UEFI, but I'd love to help! Also, for parsing executables, instead of using xmas-elf, try goblin.

[phil-opp]

Thanks for offering your help! I made some good progress over the last few days, but it's still an early prototype with lots of moving parts so I think it's difficult to collaborate at this stage. I will let you know when I finished the first prototype version, then there will probably be more opportunity to help.

Also, for parsing executables, instead of using xmas-elf, try goblin.

I worked with goblin before and it works quite well too, but I found xmas-elf more useful for no_std projects. If I remember correctly, many of goblin's abstractions are not available without the standard and alloc libraries.

[phil-opp]

Copying my status update from our gitter chat:

As a short update from my side, I got the UEFI bootloader working and integrated with the BIOS bootloader. The code is available here: https://github.com/rust-osdev/bootloader/tree/uefi

I added a new builder binary that you can use to build both the BIOS and UEFI variants of the bootloader. It's aliased to cargo builder in the .cargo/config.toml. The basic usage is:

cargo builder --kernel-binary /path/to/your/kernel/binary --kernel-manifest /path/to/your/Cargo.toml --firmware uefi

For the BIOS variant, the output of this is the disk image, so you no longer need bootimage for that. For the UEFI variant, I currently only create the .efi file, but I plan to create the fat32 image at some point too. Using OVMF, you can run the .efi file directly in QEMU.

In order to fully get rid of bootimage, I created two helper crates : bootloader_locator for locating a bootloader dependency in the dependency tree and runner_utils for detecting whether an executable is a test binary and providing a run_with_timeout function that can be used for running tests.

Using these crates, it's relatively easy to write to write an own create_disk_image builder function for our kernel: https://github.com/phil-opp/blog_os/blob/db307d89bc9e52bc0ab5eb7faf28f0d6d6637b93/disk_image/src/lib.rs . Creating an own runner executable is also possible in a few lines: https://github.com/phil-opp/blog_os/blob/db307d89bc9e52bc0ab5eb7faf28f0d6d6637b93/disk_image/src/bin/runner.rs .

I hope that letting the users create the builder and runner executables themselves instead of relying on bootimage will make the build process less magic. Another advantage is that no globally-installed tool is needed anymore, so that all project dependencies are compiled directly with the project. You no longer have to keep the installed bootimage version in sync when you upgrade the bootloader crate to a new version. Also, working with fork of the bootloader is now easier too, since you directly talk to its build interface instead of requiring bootimage as a mediator.

Unfortunately, there is also a disadvantage: Since .cargo/config.toml values apply to all subdirectories, it is no longer possible to set a default target or automatically enable the -Zbuild-std parameter right now because these setting would also apply for the builder executable of the kernel, which is stored in a subfolder. To fix this, I proposed a --ignore-local-config flag for cargo in rust-lang/cargo#8643, which would allow us to keep the config values but ignore them for the builder/runner executables. (Let me know if you can think of an easier solution to this.)

[phil-opp]

I finally got keyboard interrupts working with the IOAPIC, which means that the UEFI prototype now has all the features of the existing BIOS implementation :tada:.

As mentioned above, I had to switch from the VGA text buffer to a pixel based framebuffer because the former is not supported on UEFI. To keep things consistent, I also changed the BIOS implementation in the same way. The second difference is that we're no longer using the legacy PIC for hardware interrupts, since it is not supported either. Instead, we set up the local APIC and IOAPIC to do things properly.

The result looks like this for UEFI:

For BIOS systems, it should be exactly the same:

(There are only some differences in the default values of some registers.)

This means that we don't need to do anything special for either UEFI or BIOS from the kernel side. The new boot info structure provides a common interface that works with both firmware variants.

The next step is to merge and publish the new bootloader implementation (after some polishing). Then I'll update and restructure the blog for the new implementation, including rewriting some posts ("VGA Text Buffer" and "Hardware Interrupts"). Since the IOAPIC requires allocations, I plan to move the new "Hardware Interrupt" post after the memory management posts.

[phil-opp]

@64 I created a small apic crate for accessing the most important registers of the local APIC and the IOAPIC. You mentioned above that you also wanted to create an apic crate. Do you want to build your own or would you consider collaborating on this under the rust-osdev organization, e.g. starting from my prototype? I don't have my code online yet, put I plan to push it in the next few days. If you're interested in collaborating, I could ping you after pushing it.

[phil-opp]

@64 I pushed my implementation to https://github.com/phil-opp/apic.

[64]

@phil-opp I’m happy to collaborate - putting it on the rust-osdev organisation sounds like a good idea.

I’ve given you owner access to the apic crate. Feel free to nuke what’s there and replace it with your prototype (my code is just a stub atm).

[phil-opp]

Awesome, thanks a lot! I moved my repo to https://github.com/rust-osdev/apic and added you to new apic team with access to that repo. Feel free to extend or change my implementation if you like, it is just a first prototype.

[Abh15h3k]

is this still being worked on?

[kennystrawnmusic]

is this still being worked on?

Yes.

[sn0n]

I know it's not September yet for the annual check-in, but I'm enjoying the series thus far!!

[npetrangelo]

Could there be a way to easily configure USB I/O if you’re using an external machine to display stuff? Like if I ran the OS on a raspberry pi and wired it to my laptop to see its logs?

[bjorn3]

After exitting boot services (which you have to do before you can use many features a proper PS needs), the firmware will no longer provide any help for that. You did have to write a USB stack and USB serial driver yourself. Something easier if you got another raspberry pi or something like that would be to connect the UART pins between the raspberry pi and the other device. This would allow you to read whatever is sent by the OS through the UART (which is much easier to write a driver for) on the other device and send it back to your computer. Make sure to connect the rx pin on one side with the tx on the other side and vice versa. Also you have to be very careful that both sides use the same voltage for the UART. For example I believe an Arduino uses a higher voltage than a Raspberry PI, which can damage the Raspberry PI.