UEFI?

[johalun]

Hi! Thanks for this great blog series. It's really awesome and helping me a lot in learning how an OS works.

Do you have any plans of adding support for UEFI boot? I assume the assembly would be much simpler since UEFI setup alot of that for you, including preparing a graphical framebuffer to draw on.

As for the UEFI boot loader, I assume grub-efi can be used to load a kernel from a fat/ext/ufs partition. Or, perhaps the kernel can be put on the ESP and loaded directly by the UEFI BIOS.

If I find the time, maybe I'll play around with it and do a PR. Just wanted to make sure we're not doing double work..

[le-jzr]

Booting on UEFI via grub doesn't really give you anything special. But as it happens, I'm working on a pure UEFI kernel, beginnings of which I'll upload to GitHub soon. (If you wanna try on your own, I found it best to duplicate gnu-efi build process. Their README was very helpful.)

[le-jzr]

Sooo, I uploaded what I have now. https://github.com/le-jzr/sisyphos-kernel-uefi-x86_64

It's not much, but it builds, runs and prints stuff. Everything else is work in progress.

[johalun]

Will try it out! Thanks for sharing :)

[phil-opp]

Cool! I'm planning a second edition of the tutorial with an own bootloader, since grub causes problems on many architectures. Ideally, we would support both UEFI and legacy BIOS and provide tutorials for both systems. So thanks a lot for sharing your prototype @le-jzr!

I try to create a clean branch for the second edition in the next days. I think I will focus on the legacy bootloader first, but I'll be happy to merge any UEFI related PR.

[gil0mendes]

@phil-opp Writing a bootloader for this series would be a great improvement 👏🏻

In times I wrote my own bootloader, but is in C and I want to convert it to Rust, but for now, I'm without time for that. :(

[toothbrush7777777]

I have been working on a UEFI kernel written in Rust, although it could equally be used to load a kernel and there hasn't been any need for any x86 Assembly except for halting the processor.

If you like, I can upload my code and share a link when the code is more presentable (I'm learning Rust and come from a VB6/VB.NET/JavaScript background).

[phil-opp]

@toothbrush7777777 That would be awesome!

[toothbrush7777777]

@phil-opp @johalun I have uploaded my working example UEFI application to https://github.com/toothbrush7777777/uefi-app-x64.

Sorry for the really long delay. I've been very busy with work and travels. 😭

[phil-opp]

@toothbrush7777777 That's awesome! I just tried it shortly and it compiled without problems. The code looks really short, that's great! I didn't manage to run it yet, but I will try again tomorrow.

[toothbrush7777777]

@phil-opp I’m glad you like it. The most complicated part is the code for converting to UTF-16 and printing in chunks. I haven’t finished the project yet, but it at least prints and exits correctly.

[toothbrush7777777]

@phil-opp The code doesn’t require LLVM anymore! I merged a pull request from @kkk669 to use the lld-link distributed with recent toolchain installs of Rust nightly.

[phil-opp]

Hey @toothbrush7777777, sorry that I haven't replied for so long! I was busy with the BIOS bootloader and the bootimage tool.

I'm now thinking about the best way to intregrate UEFI booting into the blog and/or bootimage. I think we have the following options:

• Build a plain UEFI app, maybe with an library that provides abstractions for common functions (loading a file, retrieving the memory map, etc.). The users have to set up and mount the disk image on their own.
• Add UEFI support to bootimage, so that the disk image is automatically created from the kernel executable. This would be more easy for the user, but it might be lots of work (creating the file system etc.).
• Add UEFI support to bootimage (see previous point) and create an "UEFI bootloader" with the same behavior and interface as the BIOS bootloader. This means that the user continues to have an ELF executable with GNU calling convention, a nice memory map abstraction, and already a correct page table mapping for their kernel (UEFI only does identity mapping AFAIK).

Option 3 might fit best into the blog and allows the user to start with interesting things (e.g. input and output), but it would mean a lot work for us and probably too much magic for some people (albeit this is solvable by writing additional blog posts about the lower abstraction layers).

What do you think?

[toothbrush7777777]

@phil-opp Option 3 sounds good. Setting up the disk and file-system is a lot of work, though. Someone would need to write libraries to create/edit MBR and GPT partitions, and FAT32 file-systems. This would be similar to how the bootimage tool already works.

I won't be able to help much for a few months, though. I'm very busy with work right now.

[reynoldsbd]

Hello! I have been working on a UEFI abstraction crate for Rust. It follows the UEFI spec closely and could be used to implement any of the three options listed. It also contains an example Makefile that shows how to build and run a UEFI-based application under QEMU.

My next goal for this crate is to add support for file I/O. Loading a file from disk in a UEFI environment is probably not so difficult. Section 13 of the UEFI spec describes the file system protocols that must be implemented by compliant UEFI firmware. These make it possible to do file I/O directly, without needing to implement any block device or file system drivers.

A UEFI-based version of bootimage's bootloader would just need to load the user's ELF file into memory from the EFI system partition, call UEFI's ExitBootServices routine, and then jump to the user's entry point.

[reynoldsbd]

I think I have a working implementation here, but there is some bad news. Evidently, VGA is not supported when booting under UEFI. This thread has some further context.

In practical terms, this means that although the kernel can be loaded and executed under a UEFI system, it won't be able to access VGA hardware to output text. At best, the VGA driver will just be writing bytes to an unimportant memory region.

If you still wish to support UEFI, I see two options:

1. Instead of using VGA to output text from the OS, use serial port I/O. This should work regardless of whether the system is booted under BIOS or UEFI (although I have not yet tested it), so you won't need to diverge any blog posts. The downside, of course, is that you'd need to completely replace the "VGA Text Mode" post with an analogous post about serial I/O, as well as modify the "minimal kernel" post to output "Hello world!" to serial.
2. Write an "efifb" driver for outputting text from the kernel. This could either be as an alternative to the "VGA Text Mode" blog post, or as a "magic" support crate that users just import and use. In either case, such a driver would be much more complex than a VGA text-mode driver, because it would need to render fonts directly into a linear frame buffer. To accomplish this, we might want to reference another implementation, such as OpenBSD's efifb driver.

All things considered, I think option 1 involves less work, and I'd be happy to assist with the changes.

[phil-opp]

@reynoldsbd Thanks a lot for your work! Sorry for replying so late.

Evidently, VGA is not supported when booting under UEFI.

That's a pity. Serial I/O is a good alternative, but only for emulation. Most real systems don't have a serial port anymore, so writing to a serial ports wouldn't have any observable effect. I haven't had the time to read up on UEFI, but isn't there some way to output some text on the EFI console?

I'm unsure how we should integrate UEFI into the blog, but I definitively plan to do it. It seems like the "support both BIOS and UEFI transparently from bootimage" approach doesn't quite work out. Making the blog UEFI-first would be cool, but too radical in my opinion for various reasons (there are still people with old BIOS systems, I don't have much experience with UEFI yet, emulation of UEFI is more difficult, cross-platform tooling does not exist yet, etc).

So I think the best way forward is to provide optional, non-transparent UEFI support: The bootimage tool gets support for building UEFI disk images from Windows executables and we add a post that explains how to transform the BIOS kernel into a hybrid BIOS/UEFI kernel. In this post the user learns how to use conditional compilation based on the target triple to select either the VGA text buffer or the EFI console for screen output. What do you think? I'm open to alternative ideas of course.

[reynoldsbd]

UEFI does support console I/O while "boot services" are active. The catch is that the bootloader/OS must disable those boot services (by calling ExitBootServices) before it is allowed to take full control of the system (i.e. manage page tables, setup interrupts, or directly access any hardware). After that, there doesn't seem to be any alternative for getting text to the screen except by manually rendering it.

I agree, it sounds like there's no great way to transparently support both boot methods. I think we would need the following:

• a post about adding support for UEFI boot via conditional compilation (probably via a feature flag, as the target triple would likely remain unchanged), with the caveat that screen output will not work
• bootimage support for building a UEFI-bootable image
• an LFB-based console output driver (no small feat...) and a corresponding post? (possibly an advanced topic)

I still think there may be value in adding a serial-based console output driver, if for no other reason than to make UEFI more accessible to a beginner (as the LFB-based driver is likely to be quite complex).

[IsaacWoods]

I've found serial invaluable for debugging even with a working VGA driver (mainly because you aren't restricted to 25 lines), so that could be a valuable post (it also isn't too tricky, if a little bit archaic).

On the LFB-based driver, we could always write a "black-box" crate, sort of like how x86_64 is used without being implemented in the blog, but is well documented. As a beginner to OSdev / Rust, writing one from scratch might seem a bit disconcerting.

[toothbrush7777777]

@reynoldsbd Then why don’t you set up the VGA buffer before (or after, with VBE) calling ExitBootServices? I seem to remember it being possible with one of the UEFI protocols.

[reynoldsbd]

@toothbrush7777777 there is a "Graphics Output Protocol" (GOP) that provides access to linear frame buffers, but these buffers correspond to pixels, not characters. AFAICT there is just no such thing as VGA or VBE when booting under UEFI, or at least none that is guaranteed to be available by the specification :/

[le-jzr]

Although not as simple as letting UEFI draw the glyphs, writing text to a linear framebuffer is still fairly simple, as long as you stick to a bitmap font. Hardcoding the ASCII set into an array of small bitmaps is a good start (for 8x8 glyphs, you can easily represent each glyph as u64). For larger things, one could use the GNU Unifont if one can swallow the GPL licensing.

[GabrielMajeri]

@phil-opp I'm the author of (yet another?) Rust UEFI crate (uefi-rs). I don't claim to be an expert, but it's proven useful already in some people's hobby kernels. I'm interested in collaborating with you to get UEFI support for this awesome guide.

Plan

My idea for adding UEFI support would be something like this: we just replace the _start function with an uefi_start function (which will take in a UEFI image Handle and a reference to the SystemTable), and completly throw away the bootloader. Yes, really, people don't even have to worry what a boot loader is. People have two options of running the generated .EFI applications / kernels:

• put them on any USB stick and run them on real computers (boot the file from an UEFI shell, or move the file to /efi/boot/bootx64.efi to boot it automatically)
• boot them into QEMU with OVMF directly (you can tell QEMU to boot from a local directory as-if it were a FAT drive).

UEFI starts the binary in 32-bit/64-bit mode with flat paging, and GDTs are all set up. People can get a basic kernel working from Step 1, so they can play around with whatever they like.

VGA and text

As for text output, there's no need to manage the VGA text buffer anymore. The way I recommend users of my crate to do text output is to use log, and I have a simple logger which prints to UEFI's standard output.

There are many advantages of this approach: you can easily use info!() / error!() anywhere to print stuff with the right informational level, you can also modify the logger to also pipe stuff to something like a serial port, etc.

There is a massive advantage of using UEFI when it comes to graphics. VGA is legacy and hard to program, while UEFI's Graphics Output Protocol is quite high-level and is easy to use (see an example in the crate's tests, which draws some rectangles to the framebuffer).

Issue with UEFI

UEFI makes (newbie) programmers lazy. Since UEFI exposes APIs for accessing some simple filesystems, connecting to networks, has abstractions for accessing PCI devices, etc., some people might decide to use UEFI functions instead of writing their own drivers.

While it might seem advantageous for a beginner to use UEFI's functions, they won't learn anything unless they write their own drivers.

I'm hoping to get your opinion on this issue.

[GabrielMajeri]

Here's how it looks like now, with QEMU + OVMF for UEFI support:

You can see my progress in porting here. As you can see, not many changes, besides removing VGA.

[vlad9486]

Feel free to borrow code or entire crate uefi.

@GabrielMajeri IMHO, some people want to use UEFI as a standardized loader and perform "exit boot services" once booting is done. Osdev is about writing the own OS. Using UEFI API is not much different from using existing OS like Linux.

[GabrielMajeri]

@vlad9486

Using UEFI API is not much different from using existing OS like Linux.

I specifically said I don't recommend people use UEFI APIs besides the basics.

In a computer with UEFI support and no BIOS, without using UEFI GOP or UEFI's stdout, you have no means of outputing text. You can't use VGA: you would have to write a full graphics driver for every GPU you want to support. That means thousands of lines of code to set up a framebuffer for each GPU manufacturer. As for serial output, you can use that with QEMU, but good luck using it on a modern computer.

IMHO, some people want to use UEFI as a standardized loader and perform "exit boot services" once booting is done.

you can call ExitBootServices and keep using some of the existing protocols for standard output. Even Linux uses UEFI's GOP if you lack a driver for your GPU.

[toothbrush7777777]

@GabrielMajeri I didn’t know you could continue using GOP services after ExitBootServices. Do you know which other protocols can be used after ExitBootServices?

[GabrielMajeri]

@toothbrush7777777 The UEFI spec says that protocols which also work at runtime are marked so in their documentation.

AFAIK, the only protocols which keep working at runtime are:

• GOP, but you're not allowed to change the video mode anymore, just to use the existing framebuffer you set up.

• the network-related protocols, although that was just a recommendation, not a hard requirement.

Besides the GOP, most services transitioned to being boot time only, to discourage (ab)using them while the OS is running.

[vlad9486]

@GabrielMajeri

I specifically said I don't recommend people use UEFI APIs besides the basics.

Ok, sorry for misunderstanding, I just reply on "Issue" section.

you have no means of outputing text

Yeah, it is a problem... It is the shame that the gpu vendors could not produce standard for framebuffer access.

[phil-opp]

@GabrielMajeri Sorry for the delay, I was on vacation for the past weeks.

I haven't had the time to try it yet, but it sounds really great! The diff is much smaller than I imagined. I would be more than happy to collaborate with you to bring first-class uefi support to the blog! Thanks so much for your work!

One thing that's important to me: The guide should continue to work natively on Windows, macOS, and Linux. Is this possible with UEFI (including booting in QEMU)?

Also, I'm not 100% percent sure if we should really omit the bootloader. I like the idea of not depending on it, but the flat identity mapping done by UEFI has disadvantages to the bootloader mapping (e.g. no guard page below the stack, no support for higher half kernel). Also, we still need the bootloader for BIOS booting (which we still want to support) and I don't want to drop support for BIOS-booting.

In case we decide to keep the bootloader, we would need to add UEFI support to it. This includes adding an UEFI entry point, reading out the memory map, loading the kernel ELF file, and doing the kernel remapping. Most of these steps are already implemented, so it should be relatively little work.

In case we decide to not depend on the bootloader for UEFI, we would need to convert the bootloader to some kind of UEFI compatibility layer, so that a BIOS boot looks like an UEFI boot to the OS. This would include loading .exe binaries and providing the UEFI functions (how do they work by the way? through interrupts?). I'm not sure if this is possible, but it would be more work.

What do you think?

[phil-opp]

@GabrielMajeri Could you give me a quick overview of the steps that are needed to build your UEFI port and run it in QEMU?

Edit: I tried cargo xbuild --target x86_64-blog_os.json, but I get lots of "rust-lld: error: unknown file type: uart_16550-f460280a685f7c22.uart_16550.avtptz6d-cgu.1.rcgu.o" errors.

[GabrielMajeri]

@phil-opp

One thing that's important to me: The guide should continue to work natively on Windows, macOS, and Linux. Is this possible with UEFI (including booting in QEMU)?

Building the code should work, as long as Rust + rust-lld is installed.

QEMU + OVMF should work on all platforms, but the emulation can be somewhat slower if the system lacks support for hardware virtualization. That would probably involve something like KVM or HAXM for MacOS/Windows.

The Python script we use in the uefi-rs repo does require Unix named pipes for headless QEMU monitoring (when running unit tests), but I see no issue with running QEMU on Windows, with a GUI turned on.

Also, I'm not 100% percent sure if we should really omit the bootloader. I like the idea of not depending on it, but the flat identity mapping done by UEFI has disadvantages to the bootloader mapping.

I agree. My idea was to make it really easy for beginners to get something set up, and then introduce the bootloader in the secondary tutorials.

The README for uefi-rs does tell people to use a boot loader, or wrap the kernel.elf inside a kernel.efi self-extracting binary, if they need advanced features. Something like Linux kernel's EFISTUB.

I tried cargo xbuild --target x86_64-blog_os.json

Strange, it works for me. Do you have the latest cargo-xbuild and nightly Rust? Also maybe try a cargo clean? What is your OS?

[IsaacWoods]

One thing that's important to me: The guide should continue to work natively on Windows, macOS, and Linux. Is this possible with UEFI (including booting in QEMU)?

One of the problems I envisage is a cross-platform way to create the GPT image with the FAT partition. I'm using parted on Linux, but I'm not sure what you'd use on other platforms.

I wonder if a Rust-based GPT library would be a good addition to rust-osdev to build these images from within bootimage. I had looked at this but it doesn't quite look complete enough to actually write an image with.

[toothbrush7777777]

@IsaacWoods A comprehensive library to read and write/modify GPT headers (preferably no-std and using byte arrays or at least generic in terms of I/O) would be excellent. So would a good FAT library — though perhaps fatfs would work.

[reynoldsbd]

@phil-opp For what it’s worth, the Windows Subsystem for Linux has grown quite mature. In most cases, Windows users can now follow the Linux instructions completely verbatim.

Would it be realistic to revise your platform compatibility requirements as follows: “The guide should continue to work natively on macOS and Linux (including WSL).”

This has some distinct benefits:

• You no longer need to maintain compatibility with Windows directly, provided the Linux instructions still work under WSL
• we can use a pre-built OVMF package for booting UEFI code under QEMU
• we can use Linux tools like mkfs and xorriso to build a bootable UEFI image

[rpjohnst]

One of the problems I envisage is a cross-platform way to create the GPT image with the FAT partition.

A good stop-gap here is that QEMU allows you to pass in a directory for it to expose as an emulated EFI system partition. This can get you a long way without ever using actual disk images.

Windows Subsystem for Linux

WSL is not Windows, and it's really quite annoying to ask people to use it for something that has no reason to be Linux-specific. It's not something your average Windows install is ever going to have set up, either, so it's only slightly less of a pain than just using a VM.

[GabrielMajeri]

a cross-platform way to create the GPT image

You don't need to make a GPT partition (unless you plan to boot from USB on a real computer)! As it was mentioned above, using directories as emulated FAT partitions works perfectly fine. That's how I was developing for UEFI before I moved to Linux, and that's still how it works on Linux and on all platforms.

we can use a pre-built OVMF package for booting UEFI code under QEMU

kraxel provides pre-built OVMF binaries users can download, and we also mirror them on a branch of the repo.

[phil-opp]

@GabrielMajeri

I agree. My idea was to make it really easy for beginners to get something set up, and then introduce the bootloader in the secondary tutorials.

Sounds like a good solution.

Strange, it works for me. Do you have the latest cargo-xbuild and nightly Rust? Also maybe try a cargo clean? What is your OS?

It worked after a rustup update, so I guess my nightly was just too old. I got it booting in QEMU and it was relatively painless. The boot process was much slower, probably due to OVMF. I got a segment not present fault, which resulted in a double fault (I assume the same that you had in your screenshot). I didn't try to investigate it though.

I wonder if a Rust-based GPT library would be a good addition to rust-osdev to build these images from within bootimage

It would be a great addition, but I assume would be a lot of work?

[ms140569]

Hi *,

i'll try to sum-up a couple of things to make sure i've understood some rationals correct:

• Multi-Platform including Windows is and will be important.
• The Bias of this Blog is not only on the OS itself, but on booting, Boot-managers, Boot-loading etc. as well.

Or to put it the other way round: If you are only interested in teaching the OS part and could live with Unix/Linux/MacOS as development platform - What would you loose in creating an ELF/multiboot2 compliant OS and (UEFI)Boot it using grub2?

cheers, Matthias

[phil-opp]

@ms140569 Grub is not so easy to set up on Windows. Further, it only brings us to protected mode when doing a BIOS boot, so we need to do the switch to long mode ourselves.

The idea of the bootloader crate and the bootimage tool is that they set up an initial environment for your kernel, so that we can directly start with the interesting parts and not need to do any processor configuration before. Also, they don't require any C or system dependencies, so that they work natively on all three operating systems.

Currently the bootloader only supports BIOS booting, but the plan is to add support for UEFI and multiboot2 as well, either through cargo features or by creating an hybrid image (if possible).

[yw662]

Then why not try sth. like "efi stub" technique ? UEFI firmware will setup long mode with a flat memory map if the platform is x64. We even do not need a bootloader.

[yw662]

The kernel just need to ST.BS.GetMemoryMap, ST.BS.ExitBootServices, and then ST.RT.SetVirtualAddressMap whenever needed, since the last function is UEFI runtime function.

[phil-opp]

@yw662 There are few issues with using UEFI directly:

• The kernel only supports UEFI and not legacy BIOS booting (e.g. for older systems). To avoid this problem we could write a BIOS bootloader that simulates an UEFI environment, but this is not easy as UEFI has lots of functionality.
• UEFI creates only a basic identity mapping, so we would need to do a full remapping to improve safety of our kernel (unmap unused pages, use correct read/write/execute permissions, set up a guard page for the kernel stack, …).
• An UEFI app is not directly bootable. So there is an additional step to create a bootable disk image from the .efi file. The step involves creating a file system, which is done differently on Windows and Linux.

The bootloader+bootimage combination on the other hand abstracts away the booting details, sets up a safe page mapping for the kernel, directly outputs a bootable disk image, and works natively on Windows, macOS and Linux. For this reason I think it's a good idea to keep the bootloader even for UEFI.

[nebrelbug]

Any updates?

[phil-opp]

@nebrelbug Not yet, unfortunately. There's just so much other stuff to do.

[phil-opp]

There is a new tool to create a bootable UEFI image from .efi files and run the image in QEMU: https://github.com/richard-w/uefi-run. This seems really useful for adding UEFI support to bootloader/bootimage.

[nebrelbug]

@phil-opp that seems really nice

[phil-opp]

It seems like the 8259 PIC is not available for UEFI: https://github.com/phil-opp/blog_os/issues/480#issuecomment-480495268. So we need a different solution for this too (e.g. directly use the APIC).

[lvndry]

What's the state of this issue please ?

[IsaacWoods]

@lvndry I obviously don't know what Phil is planning to do tutorial wise, but this is further along now ecosystem wise:

• We still don't have a UEFI-based solution like bootloader is for BIOS - I think the end-goal will still be to write one that can be used easily from something like bootimage, but it still needs a lot of groundwork
• The uefi-rs crate is now pretty featureful, and can be used to create a UEFI bootloader without too much fuss. I have an example of one in my project here.
• You will need to migrate away from legacy hardware such as the i8259 PIC to more modern stuff like the APIC. Setting up the APIC isn't too difficult with the help of my acpi and aml crates. These are things that you'll probably want to do anyway as your OS matures.

[phil-opp]

I'm currently looking into adding UEFI support to the bootloader crate. I already managed to create a small example application using uefi-rs and I'm currently trying to find a common abstraction for both BIOS and UEFI booting. There are two main challenges where we will need to adjust the tutorials:

• Text Output: We currently use the VGA text mode for printing text, which is not supported with UEFI. Instead, it provides a EFI_SIMPLE_TEXT_OUPUT_PROTOCOL that can be used to print text to the screen. Unfortunately, this protocol is only available until ExitBootServices is called, so we can't use it in our kernel. (@IsaacWoods Am I understanding this correctly?)

  If I understand things correctly, only the framebuffer provided by the graphics output protocol (GOP) is available after exiting boot services. This means that in order to get a common interface, we need to make the bootloader default to a pixel-based framebuffer for both BIOS and UEFI booting. We then could report the framebuffer address and metadata (resolution, stride, pixel format, etc) in the boot info.

• Hardware Interrupts: As @IsaacWoods mentioned, it seems like the legacy PIC is not supported with UEFI, so we need to migrate the tutorial to the APIC. This is something that I wanted to do anyway because it is required for multi-core programming, but I originally planned to do it in a separate post together with enabling the other CPU cores. If the PIC really isn't supported at all with UEFI, it might make sense to instead rewrite the Hardware Interrupts post to directly use the APIC.

Note these are just some preliminary thoughts. I might very well forget or misunderstand something.