Rewrite

[rybot666]

Bootloader Rewrite

This is the tracking issue for the bootloader rewrite. We're tracking our progress under the "Rewrite" milsestone. Our progress is as follows

• [x] Successfully execute Rust in real mode
• [x] Enter protected mode
• [ ] Use virtual 8086 mode to read the kernel in chunks, copying to protected mode
• [ ] Set up paging
• [ ] Enter long mode
• [ ] Jump to the kernel

Our eventual roadmap is to include UEFI support (@phil-opp is working on that) and multiboot client support, so we can load our kernel from GRUB (and other complaint loaders)

[rybot666]

On the other hand, 16-bit Rust might be a whole other project.

[phil-opp]

Thanks for opening this issue! I absolutely want to replace as much assembly as possible with Rust, I just didn't have the time to look into this yet.

I think LLVM even has explicit support for real mode x86, so it should be possible to rewrite the 16 bit assembly in Rust.

[rybot666]

Sorry for the late reply but I'll try and look into this, and I'll make a PR. I'm away from my development machine right now, but I can still research and write some code when I get back next week.

LLVM has support for 16-bit, sort of. From a section in the LLVM User Guide:

For the X86 target, clang supports the -m16 command line argument which enables 16-bit code output. This is broadly similar to using asm(".code16gcc") with the GNU toolchain. The generated code and the ABI remains 32-bit but the assembler emits instructions appropriate for a CPU running in 16-bit mode, with address-size and operand-size prefixes to enable 32-bit addressing and operations.

[phil-opp]

It was a very fast reply!

LLVM has support for 16-bit, sort of. From the a section in the LLVM User Guide:

Awesome!

I'll try and look into this, and I'll make a PR. I'm away from my development machine right now, but I can still research and write some code when I get back next week.

Thanks a lot for tackling this! Don't hurry, I'm on vacation until the end of september anyway, so I won't be able to review PRs until then.

[phil-opp]

Triage: I'm still interested in this. There was a recent [reddit post](https://www.reddit.com/r/rust/comments/ask2v5/dos_the_final_frontier/] where someone got Rust working in a 16-bit DOS environment, which sounds exactly like what we want to do. The corresponding code is available on GitHub I assume that especially the target JSON is useful for us.

[lashtear]

Keep in mind that the 16bit x86 output of llvm (and gcc/gas for that matter) often still assumes it's a higher-word processor, and may emit 32bit and 64bit prefixes for dealing with data of that size. For example, that target JSON will work for 80386+, but probably not be operational on an 80286 or below. Side note-- yes the 286 can do protected-mode 16bit.

But yes-- worth looking at! It's entertaining that ARM et al get so much credit for having the Thumb ISA when other architectures more or less fell into that feature-set by accident.

[rybot666]

@phil-opp That target spec needs GCC, and I thought we were aiming to avoid additional software

[phil-opp]

@rybot666 We definitely want to keep using LLD, but maybe LLD has support for these (or similar) arguments too.

[rybot666]

@phil-opp Working on modifying that DOS target to rust-lld. Seems like it doesn't support the options we need (-m16, -ffreestanding). I don't know what we could do to fix this... (Although -m16 isn't needed, because we use the target i386-unknown-none-code16 that does exactly what -m16 does). Do you have something like Discord for live chat?

[phil-opp]

I'm not sure if we need -ffreestanding with LLD, we don't use it for our freestanding kernel/bootloader either. Maybe it's a GCC related flag.

Although -m16 isn't needed, because we use the target i386-unknown-none-code16 that does exactly what -m16 does

That's good to know!

Do you have something like Discord for live chat?

Yes, I'm phil-opp#1436 on discord.

[rybot666]

@phil-opp I don't know if you're free, but I'm working on this now - I sent you a friend request on Discord

[michalfita]

Do you really need to support pre-EFI machines?

[phil-opp]

@michalfita We currently only support non-EFI booting (nobody has had the time to add EFI support yet).

And yes, I think that it's worth keeping support for pre-EFI machines, since these machines are often used for testing. For example, I use my old machine for booting my kernel so that I can continue working on my main machine in parallel.

[rybot666]

So, I got Rust running in bare metal 16-bit mode. I had to use a bit of assembly to set up stack and load rust from disk. The issue I'm thinking we may have is that booting needs a lot of assembly anyway so I don't know how much can be ported to pure Rust. Also unsure as to how to get cargo to build me multiple targets and then link them together (so we can swap from our 16-bit target to a 32-bit one). I think that disk loading logic will be much easier to understand with Rust though.

cc @phil-opp

[rybot666]

It's a project to get it into as much Rust as possible, to prove that it's possible and to make it easier to explain (I'd say Rust traits and structs are easier to understand than raw mov's and int's). Anyway UEFI would be easier with rust bindings

[rybot666]

Also yeah I hit close and comment by mistake like an idiot on mobile

[phil-opp]

@rybot666 Sorry for not replying earlier!

So, I got Rust running in bare metal 16-bit mode.

Awesome! Do you have your code online somewhere so that I can take a look?

Also unsure as to how to get cargo to build me multiple targets and then link them together (so we can swap from our 16-bit target to a 32-bit one).

I think it should be possible by creating static libraries and linking them together. We might need to rename some colliding symbols, but it should be doable.

The issue I'm thinking we may have is that booting needs a lot of assembly anyway so I don't know how much can be ported to pure Rust.

I think we should be able to port a significant part of the assembly code. As you said, especially the disk loading code will definitely benefit from this.

Anyway UEFI would be easier with rust bindings

Breaking this project into 16-bit and 32-bit subprojects should also make it easier to port it to multiboot and UEFI since it allows us to reuse individual components (e.g. the kernel mapping code) while omitting others (e.g. the switch from real mode to protected mode).

[phil-opp]

@rybot666 I found your repository at https://github.com/rybot666/bootloader and used it to put together a simple "Hello from Rust" application at https://github.com/rust-osdev/bootloader/tree/16-bit/real_mode. Now we can start porting the assembly code :tada:.

[rybot666]

Awesome! I think we'll need at least a few lines of assembly to load the rust from disk, unless we use #![no_core] and put it in the bootloader, but that's very annoying and I doubt a good abstraction could be made with no standard library and only 510 bytes

[bjorn3]

When using #![no_core] you will need to provide a lot of lang items yourself. Using the panic_immediate_abort feature flag of libcore combined with -Clto, -Copt-level=s, -Ccodegen-units=1 and disabling incremental compilation may generate a small enough bin (not sure though) while requiring a lot less work.

[phil-opp]

@rybot666 I have a question about the data layout in https://github.com/rybot666/bootloader/blob/bd0bce685052cf1a4f562247a70a84ccbf64275b/rusty16.json#L4

Why did you use p:32:32? Aren't pointers just 16-bit in real mode?

[bjorn3]

Quoting https://releases.llvm.org/3.5.0/tools/clang/docs/UsersManual.html#target-specific-features-and-limitations:

The generated code and the ABI remains 32-bit but the assembler emits instructions appropriate for a CPU running in 16-bit mode, with address-size and operand-size prefixes to enable 32-bit addressing and operations.

i386-unknown-none-code16 still uses 32bit pointer. To actually access data above 0xffff requires Unreal mode though.

[phil-opp]

Makes sense, thanks!

[phil-opp]

I managed to port the disk address packet related code to Rust: https://github.com/rust-osdev/bootloader/tree/16-bit/real_mode. I did not exceed the 512 byte limit, but only by a few bytes. My next step is to try to link a second Rust binary/static lib as the second stage.

[bjorn3]

I have heard that opt-level=z can sometimes require more space than opt-level=s.

[phil-opp]

I tried both s and z with no difference.

[phil-opp]

I'm currently stuck linking two static libraries together because they both include a copy of compiler_builtins, which leads to duplicated symbols. What I'm trying to do is to build each stage as a separate staticlib in order to clearly separate the stages in the linker script. My current state is available here: https://github.com/rust-osdev/bootloader/tree/16-bit-wip/real_mode (see the build script).

Does anyone have an idea how to fix this problem? Or maybe there are other approaches that work better?

[rybot666]

Could you make a fake version with just "extern" symbols and link it to the first stage, then have a full version in second? That would reduce space, but I don't know if "extern" actually creates new symbols or just specifies something to the linker. We could also build it into it's own module and link it with a linker script, but I don't see why that would be necessary?

On Fri, 20 Dec 2019, 17:20 Philipp Oppermann, notifications@github.com wrote:

I'm currently stuck linking two static libraries together because they both include a copy of compiler_builtins, which leads to duplicated symbols. What I'm trying to do is to build each stage as a separate staticlib in order to clearly separate the stages in the linker script. My current state is available here: https://github.com/rust-osdev/bootloader/tree/16-bit-wip/real_mode (see the build script).

Does anyone have an idea how to fix this problem? Or maybe there are other approaches that work better?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rust-osdev/bootloader/issues/24?email_source=notifications&email_token=AEOYYSPHZ5VQUYWFUMBVFL3QZT5GBA5CNFSM4FRRQ6B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHNRWOA#issuecomment-568007480, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEOYYSOOESVY3JQ7KXSQEJDQZT5GBANCNFSM4FRRQ6BQ .

[phil-opp]

I want to separate the first and second stage as much as possible to enforce that no second stage function is called before its loaded.

The solution to my problem was to localize all symbols except a few selected ones using objcopy -G […].

Now my problem is linking the ELF32 files from the first stages together with an ELF64 file from later stages. Passing both to the linker results in incompatibitily errors. Trying to convert the ELF32 to an ELF64 using objcopy -I elf32-i386 -O elf64-x86-64 results in an invalid relocation type. I'm currently trying to get Rust/LLVM to directly emit an ELF64 that still contains 16-bit code, i.e. something similar to the .code16 assembly directive.

[rybot666]

Just for those who are wondering how the progress on this issue is going, we're working under the rewrite branch, and much discussion is going on in the gitter channel. Any contributions can be submitted as a PR and ideas are always welcome :smiley: