phil-opp / blog_os

Writing an OS in Rust
http://os.phil-opp.com
Apache License 2.0
15.84k stars 1.09k forks source link

UEFI? #349

Open johalun opened 7 years ago

johalun commented 7 years ago

Hi! Thanks for this great blog series. It's really awesome and helping me a lot in learning how an OS works.

Do you have any plans of adding support for UEFI boot? I assume the assembly would be much simpler since UEFI setup alot of that for you, including preparing a graphical framebuffer to draw on.

As for the UEFI boot loader, I assume grub-efi can be used to load a kernel from a fat/ext/ufs partition. Or, perhaps the kernel can be put on the ESP and loaded directly by the UEFI BIOS.

If I find the time, maybe I'll play around with it and do a PR. Just wanted to make sure we're not doing double work..

le-jzr commented 7 years ago

Booting on UEFI via grub doesn't really give you anything special. But as it happens, I'm working on a pure UEFI kernel, beginnings of which I'll upload to GitHub soon. (If you wanna try on your own, I found it best to duplicate gnu-efi build process. Their README was very helpful.)

le-jzr commented 7 years ago

Sooo, I uploaded what I have now. https://github.com/le-jzr/sisyphos-kernel-uefi-x86_64

It's not much, but it builds, runs and prints stuff. Everything else is work in progress.

johalun commented 7 years ago

Will try it out! Thanks for sharing :)

phil-opp commented 7 years ago

Cool! I'm planning a second edition of the tutorial with an own bootloader, since grub causes problems on many architectures. Ideally, we would support both UEFI and legacy BIOS and provide tutorials for both systems. So thanks a lot for sharing your prototype @le-jzr!

I try to create a clean branch for the second edition in the next days. I think I will focus on the legacy bootloader first, but I'll be happy to merge any UEFI related PR.

gil0mendes commented 7 years ago

@phil-opp Writing a bootloader for this series would be a great improvement 👏🏻

In times I wrote my own bootloader, but is in C and I want to convert it to Rust, but for now, I'm without time for that. :(

toothbrush7777777 commented 7 years ago

I have been working on a UEFI kernel written in Rust, although it could equally be used to load a kernel and there hasn't been any need for any x86 Assembly except for halting the processor.

If you like, I can upload my code and share a link when the code is more presentable (I'm learning Rust and come from a VB6/VB.NET/JavaScript background).

phil-opp commented 7 years ago

@toothbrush7777777 That would be awesome!

toothbrush7777777 commented 6 years ago

@phil-opp @johalun I have uploaded my working example UEFI application to https://github.com/toothbrush7777777/uefi-app-x64.

Sorry for the really long delay. I've been very busy with work and travels. 😭

phil-opp commented 6 years ago

@toothbrush7777777 That's awesome! I just tried it shortly and it compiled without problems. The code looks really short, that's great! I didn't manage to run it yet, but I will try again tomorrow.

toothbrush7777777 commented 6 years ago

@phil-opp I’m glad you like it. The most complicated part is the code for converting to UTF-16 and printing in chunks. I haven’t finished the project yet, but it at least prints and exits correctly.

toothbrush7777777 commented 6 years ago

@phil-opp The code doesn’t require LLVM anymore! I merged a pull request from @kkk669 to use the lld-link distributed with recent toolchain installs of Rust nightly.

phil-opp commented 6 years ago

Hey @toothbrush7777777, sorry that I haven't replied for so long! I was busy with the BIOS bootloader and the bootimage tool.

I'm now thinking about the best way to intregrate UEFI booting into the blog and/or bootimage. I think we have the following options:

Option 3 might fit best into the blog and allows the user to start with interesting things (e.g. input and output), but it would mean a lot work for us and probably too much magic for some people (albeit this is solvable by writing additional blog posts about the lower abstraction layers).

What do you think?

toothbrush7777777 commented 6 years ago

@phil-opp Option 3 sounds good. Setting up the disk and file-system is a lot of work, though. Someone would need to write libraries to create/edit MBR and GPT partitions, and FAT32 file-systems. This would be similar to how the bootimage tool already works.

I won't be able to help much for a few months, though. I'm very busy with work right now.

reynoldsbd commented 6 years ago

Hello! I have been working on a UEFI abstraction crate for Rust. It follows the UEFI spec closely and could be used to implement any of the three options listed. It also contains an example Makefile that shows how to build and run a UEFI-based application under QEMU.

My next goal for this crate is to add support for file I/O. Loading a file from disk in a UEFI environment is probably not so difficult. Section 13 of the UEFI spec describes the file system protocols that must be implemented by compliant UEFI firmware. These make it possible to do file I/O directly, without needing to implement any block device or file system drivers.

A UEFI-based version of bootimage's bootloader would just need to load the user's ELF file into memory from the EFI system partition, call UEFI's ExitBootServices routine, and then jump to the user's entry point.

reynoldsbd commented 6 years ago

I think I have a working implementation here, but there is some bad news. Evidently, VGA is not supported when booting under UEFI. This thread has some further context.

In practical terms, this means that although the kernel can be loaded and executed under a UEFI system, it won't be able to access VGA hardware to output text. At best, the VGA driver will just be writing bytes to an unimportant memory region.

If you still wish to support UEFI, I see two options:

  1. Instead of using VGA to output text from the OS, use serial port I/O. This should work regardless of whether the system is booted under BIOS or UEFI (although I have not yet tested it), so you won't need to diverge any blog posts. The downside, of course, is that you'd need to completely replace the "VGA Text Mode" post with an analogous post about serial I/O, as well as modify the "minimal kernel" post to output "Hello world!" to serial.
  2. Write an "efifb" driver for outputting text from the kernel. This could either be as an alternative to the "VGA Text Mode" blog post, or as a "magic" support crate that users just import and use. In either case, such a driver would be much more complex than a VGA text-mode driver, because it would need to render fonts directly into a linear frame buffer. To accomplish this, we might want to reference another implementation, such as OpenBSD's efifb driver.

All things considered, I think option 1 involves less work, and I'd be happy to assist with the changes.

phil-opp commented 6 years ago

@reynoldsbd Thanks a lot for your work! Sorry for replying so late.

Evidently, VGA is not supported when booting under UEFI.

That's a pity. Serial I/O is a good alternative, but only for emulation. Most real systems don't have a serial port anymore, so writing to a serial ports wouldn't have any observable effect. I haven't had the time to read up on UEFI, but isn't there some way to output some text on the EFI console?

I'm unsure how we should integrate UEFI into the blog, but I definitively plan to do it. It seems like the "support both BIOS and UEFI transparently from bootimage" approach doesn't quite work out. Making the blog UEFI-first would be cool, but too radical in my opinion for various reasons (there are still people with old BIOS systems, I don't have much experience with UEFI yet, emulation of UEFI is more difficult, cross-platform tooling does not exist yet, etc).

So I think the best way forward is to provide optional, non-transparent UEFI support: The bootimage tool gets support for building UEFI disk images from Windows executables and we add a post that explains how to transform the BIOS kernel into a hybrid BIOS/UEFI kernel. In this post the user learns how to use conditional compilation based on the target triple to select either the VGA text buffer or the EFI console for screen output. What do you think? I'm open to alternative ideas of course.

reynoldsbd commented 6 years ago

UEFI does support console I/O while "boot services" are active. The catch is that the bootloader/OS must disable those boot services (by calling ExitBootServices) before it is allowed to take full control of the system (i.e. manage page tables, setup interrupts, or directly access any hardware). After that, there doesn't seem to be any alternative for getting text to the screen except by manually rendering it.

I agree, it sounds like there's no great way to transparently support both boot methods. I think we would need the following:

I still think there may be value in adding a serial-based console output driver, if for no other reason than to make UEFI more accessible to a beginner (as the LFB-based driver is likely to be quite complex).

IsaacWoods commented 6 years ago

I've found serial invaluable for debugging even with a working VGA driver (mainly because you aren't restricted to 25 lines), so that could be a valuable post (it also isn't too tricky, if a little bit archaic).

On the LFB-based driver, we could always write a "black-box" crate, sort of like how x86_64 is used without being implemented in the blog, but is well documented. As a beginner to OSdev / Rust, writing one from scratch might seem a bit disconcerting.

toothbrush7777777 commented 6 years ago

@reynoldsbd Then why don’t you set up the VGA buffer before (or after, with VBE) calling ExitBootServices? I seem to remember it being possible with one of the UEFI protocols.

reynoldsbd commented 6 years ago

@toothbrush7777777 there is a "Graphics Output Protocol" (GOP) that provides access to linear frame buffers, but these buffers correspond to pixels, not characters. AFAICT there is just no such thing as VGA or VBE when booting under UEFI, or at least none that is guaranteed to be available by the specification :/

le-jzr commented 6 years ago

Although not as simple as letting UEFI draw the glyphs, writing text to a linear framebuffer is still fairly simple, as long as you stick to a bitmap font. Hardcoding the ASCII set into an array of small bitmaps is a good start (for 8x8 glyphs, you can easily represent each glyph as u64). For larger things, one could use the GNU Unifont if one can swallow the GPL licensing.

GabrielMajeri commented 6 years ago

@phil-opp I'm the author of (yet another?) Rust UEFI crate (uefi-rs). I don't claim to be an expert, but it's proven useful already in some people's hobby kernels. I'm interested in collaborating with you to get UEFI support for this awesome guide.

Plan

My idea for adding UEFI support would be something like this: we just replace the _start function with an uefi_start function (which will take in a UEFI image Handle and a reference to the SystemTable), and completly throw away the bootloader. Yes, really, people don't even have to worry what a boot loader is. People have two options of running the generated .EFI applications / kernels:

UEFI starts the binary in 32-bit/64-bit mode with flat paging, and GDTs are all set up. People can get a basic kernel working from Step 1, so they can play around with whatever they like.

VGA and text

As for text output, there's no need to manage the VGA text buffer anymore. The way I recommend users of my crate to do text output is to use log, and I have a simple logger which prints to UEFI's standard output.

There are many advantages of this approach: you can easily use info!() / error!() anywhere to print stuff with the right informational level, you can also modify the logger to also pipe stuff to something like a serial port, etc.

There is a massive advantage of using UEFI when it comes to graphics. VGA is legacy and hard to program, while UEFI's Graphics Output Protocol is quite high-level and is easy to use (see an example in the crate's tests, which draws some rectangles to the framebuffer).

Issue with UEFI

UEFI makes (newbie) programmers lazy. Since UEFI exposes APIs for accessing some simple filesystems, connecting to networks, has abstractions for accessing PCI devices, etc., some people might decide to use UEFI functions instead of writing their own drivers.

While it might seem advantageous for a beginner to use UEFI's functions, they won't learn anything unless they write their own drivers.

I'm hoping to get your opinion on this issue.

GabrielMajeri commented 6 years ago

Here's how it looks like now, with QEMU + OVMF for UEFI support: image

You can see my progress in porting here. As you can see, not many changes, besides removing VGA.

vlad9486 commented 6 years ago

Feel free to borrow code or entire crate uefi.

@GabrielMajeri IMHO, some people want to use UEFI as a standardized loader and perform "exit boot services" once booting is done. Osdev is about writing the own OS. Using UEFI API is not much different from using existing OS like Linux.

GabrielMajeri commented 6 years ago

@vlad9486

Using UEFI API is not much different from using existing OS like Linux.

I specifically said I don't recommend people use UEFI APIs besides the basics.

In a computer with UEFI support and no BIOS, without using UEFI GOP or UEFI's stdout, you have no means of outputing text. You can't use VGA: you would have to write a full graphics driver for every GPU you want to support. That means thousands of lines of code to set up a framebuffer for each GPU manufacturer. As for serial output, you can use that with QEMU, but good luck using it on a modern computer.

IMHO, some people want to use UEFI as a standardized loader and perform "exit boot services" once booting is done.

you can call ExitBootServices and keep using some of the existing protocols for standard output. Even Linux uses UEFI's GOP if you lack a driver for your GPU.

toothbrush7777777 commented 6 years ago

@GabrielMajeri I didn’t know you could continue using GOP services after ExitBootServices. Do you know which other protocols can be used after ExitBootServices?

GabrielMajeri commented 6 years ago

@toothbrush7777777 The UEFI spec says that protocols which also work at runtime are marked so in their documentation.

AFAIK, the only protocols which keep working at runtime are:

Besides the GOP, most services transitioned to being boot time only, to discourage (ab)using them while the OS is running.

vlad9486 commented 6 years ago

@GabrielMajeri

I specifically said I don't recommend people use UEFI APIs besides the basics.

Ok, sorry for misunderstanding, I just reply on "Issue" section.

you have no means of outputing text

Yeah, it is a problem... It is the shame that the gpu vendors could not produce standard for framebuffer access.

phil-opp commented 6 years ago

@GabrielMajeri Sorry for the delay, I was on vacation for the past weeks.

I haven't had the time to try it yet, but it sounds really great! The diff is much smaller than I imagined. I would be more than happy to collaborate with you to bring first-class uefi support to the blog! Thanks so much for your work!

One thing that's important to me: The guide should continue to work natively on Windows, macOS, and Linux. Is this possible with UEFI (including booting in QEMU)?

Also, I'm not 100% percent sure if we should really omit the bootloader. I like the idea of not depending on it, but the flat identity mapping done by UEFI has disadvantages to the bootloader mapping (e.g. no guard page below the stack, no support for higher half kernel). Also, we still need the bootloader for BIOS booting (which we still want to support) and I don't want to drop support for BIOS-booting.

In case we decide to keep the bootloader, we would need to add UEFI support to it. This includes adding an UEFI entry point, reading out the memory map, loading the kernel ELF file, and doing the kernel remapping. Most of these steps are already implemented, so it should be relatively little work.

In case we decide to not depend on the bootloader for UEFI, we would need to convert the bootloader to some kind of UEFI compatibility layer, so that a BIOS boot looks like an UEFI boot to the OS. This would include loading .exe binaries and providing the UEFI functions (how do they work by the way? through interrupts?). I'm not sure if this is possible, but it would be more work.

What do you think?

phil-opp commented 6 years ago

@GabrielMajeri Could you give me a quick overview of the steps that are needed to build your UEFI port and run it in QEMU?

Edit: I tried cargo xbuild --target x86_64-blog_os.json, but I get lots of "rust-lld: error: unknown file type: uart_16550-f460280a685f7c22.uart_16550.avtptz6d-cgu.1.rcgu.o" errors.

GabrielMajeri commented 6 years ago

@phil-opp

One thing that's important to me: The guide should continue to work natively on Windows, macOS, and Linux. Is this possible with UEFI (including booting in QEMU)?

Building the code should work, as long as Rust + rust-lld is installed.

QEMU + OVMF should work on all platforms, but the emulation can be somewhat slower if the system lacks support for hardware virtualization. That would probably involve something like KVM or HAXM for MacOS/Windows.

The Python script we use in the uefi-rs repo does require Unix named pipes for headless QEMU monitoring (when running unit tests), but I see no issue with running QEMU on Windows, with a GUI turned on.

Also, I'm not 100% percent sure if we should really omit the bootloader. I like the idea of not depending on it, but the flat identity mapping done by UEFI has disadvantages to the bootloader mapping.

I agree. My idea was to make it really easy for beginners to get something set up, and then introduce the bootloader in the secondary tutorials.

The README for uefi-rs does tell people to use a boot loader, or wrap the kernel.elf inside a kernel.efi self-extracting binary, if they need advanced features. Something like Linux kernel's EFISTUB.

I tried cargo xbuild --target x86_64-blog_os.json

Strange, it works for me. Do you have the latest cargo-xbuild and nightly Rust? Also maybe try a cargo clean? What is your OS?

IsaacWoods commented 6 years ago

One thing that's important to me: The guide should continue to work natively on Windows, macOS, and Linux. Is this possible with UEFI (including booting in QEMU)?

One of the problems I envisage is a cross-platform way to create the GPT image with the FAT partition. I'm using parted on Linux, but I'm not sure what you'd use on other platforms.

I wonder if a Rust-based GPT library would be a good addition to rust-osdev to build these images from within bootimage. I had looked at this but it doesn't quite look complete enough to actually write an image with.

toothbrush7777777 commented 6 years ago

@IsaacWoods A comprehensive library to read and write/modify GPT headers (preferably no-std and using byte arrays or at least generic in terms of I/O) would be excellent. So would a good FAT library — though perhaps fatfs would work.

reynoldsbd commented 6 years ago

@phil-opp For what it’s worth, the Windows Subsystem for Linux has grown quite mature. In most cases, Windows users can now follow the Linux instructions completely verbatim.

Would it be realistic to revise your platform compatibility requirements as follows: “The guide should continue to work natively on macOS and Linux (including WSL).”

This has some distinct benefits:

rpjohnst commented 6 years ago

One of the problems I envisage is a cross-platform way to create the GPT image with the FAT partition.

A good stop-gap here is that QEMU allows you to pass in a directory for it to expose as an emulated EFI system partition. This can get you a long way without ever using actual disk images.

Windows Subsystem for Linux

WSL is not Windows, and it's really quite annoying to ask people to use it for something that has no reason to be Linux-specific. It's not something your average Windows install is ever going to have set up, either, so it's only slightly less of a pain than just using a VM.

GabrielMajeri commented 6 years ago

a cross-platform way to create the GPT image

You don't need to make a GPT partition (unless you plan to boot from USB on a real computer)! As it was mentioned above, using directories as emulated FAT partitions works perfectly fine. That's how I was developing for UEFI before I moved to Linux, and that's still how it works on Linux and on all platforms.

we can use a pre-built OVMF package for booting UEFI code under QEMU

kraxel provides pre-built OVMF binaries users can download, and we also mirror them on a branch of the repo.

phil-opp commented 6 years ago

@GabrielMajeri

I agree. My idea was to make it really easy for beginners to get something set up, and then introduce the bootloader in the secondary tutorials.

Sounds like a good solution.

Strange, it works for me. Do you have the latest cargo-xbuild and nightly Rust? Also maybe try a cargo clean? What is your OS?

It worked after a rustup update, so I guess my nightly was just too old. I got it booting in QEMU and it was relatively painless. The boot process was much slower, probably due to OVMF. I got a segment not present fault, which resulted in a double fault (I assume the same that you had in your screenshot). I didn't try to investigate it though.

I wonder if a Rust-based GPT library would be a good addition to rust-osdev to build these images from within bootimage

It would be a great addition, but I assume would be a lot of work?

ms140569 commented 6 years ago

Hi *,

i'll try to sum-up a couple of things to make sure i've understood some rationals correct:

Or to put it the other way round: If you are only interested in teaching the OS part and could live with Unix/Linux/MacOS as development platform - What would you loose in creating an ELF/multiboot2 compliant OS and (UEFI)Boot it using grub2?

cheers, Matthias

phil-opp commented 5 years ago

@ms140569 Grub is not so easy to set up on Windows. Further, it only brings us to protected mode when doing a BIOS boot, so we need to do the switch to long mode ourselves.

The idea of the bootloader crate and the bootimage tool is that they set up an initial environment for your kernel, so that we can directly start with the interesting parts and not need to do any processor configuration before. Also, they don't require any C or system dependencies, so that they work natively on all three operating systems.

Currently the bootloader only supports BIOS booting, but the plan is to add support for UEFI and multiboot2 as well, either through cargo features or by creating an hybrid image (if possible).

yw662 commented 5 years ago

Then why not try sth. like "efi stub" technique ? UEFI firmware will setup long mode with a flat memory map if the platform is x64. We even do not need a bootloader.

yw662 commented 5 years ago

The kernel just need to ST.BS.GetMemoryMap, ST.BS.ExitBootServices, and then ST.RT.SetVirtualAddressMap whenever needed, since the last function is UEFI runtime function.

phil-opp commented 5 years ago

@yw662 There are few issues with using UEFI directly:

The bootloader+bootimage combination on the other hand abstracts away the booting details, sets up a safe page mapping for the kernel, directly outputs a bootable disk image, and works natively on Windows, macOS and Linux. For this reason I think it's a good idea to keep the bootloader even for UEFI.

nebrelbug commented 5 years ago

Any updates?

phil-opp commented 5 years ago

@nebrelbug Not yet, unfortunately. There's just so much other stuff to do.

phil-opp commented 5 years ago

There is a new tool to create a bootable UEFI image from .efi files and run the image in QEMU: https://github.com/richard-w/uefi-run. This seems really useful for adding UEFI support to bootloader/bootimage.

nebrelbug commented 5 years ago

@phil-opp that seems really nice

phil-opp commented 5 years ago

It seems like the 8259 PIC is not available for UEFI: https://github.com/phil-opp/blog_os/issues/480#issuecomment-480495268. So we need a different solution for this too (e.g. directly use the APIC).

lvndry commented 4 years ago

What's the state of this issue please ?

IsaacWoods commented 4 years ago

@lvndry I obviously don't know what Phil is planning to do tutorial wise, but this is further along now ecosystem wise:

phil-opp commented 4 years ago

I'm currently looking into adding UEFI support to the bootloader crate. I already managed to create a small example application using uefi-rs and I'm currently trying to find a common abstraction for both BIOS and UEFI booting. There are two main challenges where we will need to adjust the tutorials:

Note these are just some preliminary thoughts. I might very well forget or misunderstand something.