Build some OS kernel with mold

[rui314]

We would like to implement an alternative mechanism for the linker script language. In order to do that, we first have to learn about the existing usages of the linker script. Trying to build an OS kernel with mold should be a good way to learn about it.

[kleinesfilmroellchen]

SerenityOS x86 Kernel linker script :^)

EDIT: FYI, from SerenityOS #kernel discussions:

awesomekling: it's very likely the linker script is full of unnecessary nonsense it's organically grown without ever really having a good pruning

[rui314]

It may actually be a good test target.

[qookei]

managarm linker scripts: prekernel (eir): x86 generic 32-bit, aarch64 raspi4, riscv64 allwinner d1 - These set up an executable image intended to be loaded at a fixed point in RAM depending on the platform.

main kernel (thor): x86_64, aarch64 - These set up an higher-half kernel (-mcmodel=kernel on x86_64) that doesn't depend on physical RAM addresses. Virtual mapping is set up by the prekernel based on the ELF file of the kernel.

[qookei]

Also, not strictly an OS, but an embedded MCU SDK that uses features like LOADADDR, custom memory regions, etc.: https://github.com/raspberrypi/pico-sdk/blob/master/src/rp2_common/pico_standard_link/memmap_default.ld

[X547]

Haiku kernel for riscv64 and x86_64 can be linked with Mold after some adaptations of build system (avoiding use of ld -r that Haiku use a lot to produce intermediate compile results). Use of linker script can be avoided for this targets.

[pitust]

If i remember correctly, i was able to link one of my OSes, xtrix, using mold. In practice, however, the amount of stuff you need a linker script for is limited, and amount to setting the load address, section alignment, and creating symbols at starts and ends.

I was thinking of linking my current OS, ixOS (which uses macho, so it would be nice to have a linux-compatible linker) using ld64.mold, however it is missing some of these flags (-static, -image_base, -segalign). If those flags were implemented, though, very little linker script would be needed. The -preload flag could also be helpful (and i USED to use it, but i dropped it because the loaderi use only gives me the ELF i wrap my kernel into).

Another common use for linker script is setting the order in which specific sections land in the binary (eg. for multiboot). This is solved by ld64 with something called symbol order files, which are also a much simpler solution than linker script, IMO.

[ArsenArsen]

fwiw, I've been working lately on trying to build a kernel devoid of the need for linker scripts (not because of the performance benefits at link time, but because of the performance benefits at dev time ;) and messing about compiler flags. My test case uses the normal linker script for i686-elf -fPIC (which I picked because it's the default, and, again, I'm only worrying about convenience here) with one caveat: .mb2 is placed first, right at __SIZEOF_HEADERS__. Most custom linker scripts I've seen in the wild have similar goals: specify section order and, potentially, provide extra start/stop symbols. Both of these could be emulated without the rest of the complexity of linker scripts.

As a test case, I did a little refactoring to get my entire executable to be <32KiB in size (the GRUB2 multiboot2 header search size), linking this with mold, without a linker script, produces a binary that prints a hello world just fine (and even retains stuff like .eh_frame, etc, so exceptions do work with it). Obviously, hoping that an executable is small enough is not great, but it was the quickest way to get a test done in my case.

Anyway - I think the vast majority of cases can be addressed by adding an option to force output section order/position in the resulting file, plus a way to specify __start_/__stop_ symbols for non-identifier-representable section names, plus the physical and virtual base addresses of the whole executable and some individual segments, though I might be forgetting something.

[rui314]

@ArsenArsen Thank you for sharing your experience! Your comment is in line with what I recently found by examining linker scripts of a few different operating systems. They use linker scripts to place some section at a beginning of a file or some specific order and to define some marker symbols such as __text_start.

Typically, it looks like we need three symbols, the very beginning of the image, the end of a memory-allocated segment and .bss end. We already provides __ehdr_start, __bss_start and _end, respectively. So an OS kernel may use these symbols.

I implemented https://github.com/rui314/mold/commit/4db7574e84830d28ef03071cf49a0b6b1a0a2b2c so that if you name a section multiboot (without the beginning .), it's placed right after the ELF header, which is <32 KiB from the beginning of the file. With that, can you create an OS kernel with mold?

[ArsenArsen]

Yup, that works quite well.

I'll give a more complex kernel a try, too, and let you know how it goes; a hello world built for testing language and language support library changes doesn't make very extensive use of weird link-time stuff anyway.

For reference, here's the requisite code changes https://git.sr.ht/~arsen/frello/commit/abb66e5a498479d3863cced02325302a32064b38 (note that building this might require patches that I haven't upstreamed yet in GCC, so you might have to take my word on it booting and printing that it caught a raised exception)

[rui314]

It's so nice that we can create an OS kernel without linker script boilerplate. And a good use of constinit!

[rui314]

I wrote a proposal for new command line options to support kernel/embedded programs. Please take a look if you guys are interested.

https://docs.google.com/document/d/1bVRrKsB6iMGmay1odsmS44PyUzPXgQBH83gYGMY6LWM/edit?usp=sharing

[ArsenArsen]

I like the ideas in this document, they should be sufficient (at least for my use-cases).

As a little progress report on getting a more complex kernel running, I tried getting Managarm (of course ;) linking and booting with mold, but I ran into the issue that our prekernel doesn't handle non-page-aligned phdrs, even if they have matching attributes. I ran out of time to implement this edge case, but will try tonight or tomorrow. So far, even the existing options (with the multiboot section patch) were enough to get the prekernel booting. Here's my wip branch, though, note that it is out of date compared to upstream, and probably lacks some critical recent fixes that will break userspace - I put off fixing these since it wouldn't help me test out mold much yet).

[rui314]

@ArsenArsen I wonder if you can use -z separate-loadable-segments which tells the linker to align all loadable segments to page boundaries.

[ArsenArsen]

That still generates nonaligned LOADs sometimes:

[i] ~/managarm/build$ eu-readelf --segments test/managarm/thor 
Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  PHDR           0x000040 0xffffffff80000040 0xffffffff80000040 0x0001f8 0x0001f8 R   0x8
  INTERP         0x000238 0xffffffff80000238 0xffffffff80000238 0x00001b 0x00001b R   0x1
    [Requesting program interpreter: /lib/x86_64-managarm/ld.so]
  LOAD           0x000000 0xffffffff80000000 0xffffffff80000000 0x01cfc6 0x01cfc6 R   0x1000
  LOAD           0x01cfd0 0xffffffff8001dfd0 0xffffffff8001dfd0 0x14736e 0x14736e R E 0x1000
  LOAD           0x164340 0xffffffff80166340 0xffffffff80166340 0x000078 0x000cc0 RW  0x1000
  LOAD           0x1643b8 0xffffffff801673b8 0xffffffff801673b8 0x000958 0x048e8a RW  0x1000
  GNU_EH_FRAME   0x000eec 0xffffffff80000eec 0xffffffff80000eec 0x0000a4 0x0000a4 R   0x4
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x1
  GNU_RELRO      0x164340 0xffffffff80166340 0xffffffff80166340 0x000078 0x000cc0 R   0x1

 Section to Segment mapping:
  Segment Sections...
   00     
   01      [RO: .interp]
   02      [RO: .interp .eh_frame .eh_frame_hdr .rodata .rodata.cst .rodata.str]
   03      [RO: .text _text_stubs]
   04      [RELRO: .init_array .got .relro_padding]
   05      .got.plt .data .bss
   06      [RO: .eh_frame_hdr]
   07     
   08      [RELRO: .init_array .got .relro_padding]
[i] ~/managarm/build$ 

This should normally be okay, but the loader code takes some shortcuts, which is what I meant to change when I get time next.

[rui314]

-z separate-loadable-segments works for me, so I wonder what is different in your environment. Could you copy-n-paste your final linker invocation command line?

[ArsenArsen]

Sure,

 "/var/lib/managarm-buildenv/build/tools/mold/bin/x86_64-managarm-kernel-ld.mold" --sysroot=/var/lib/managarm-buildenv/build/system-root --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib/x86_64-managarm/ld.so -o kernel/thor/thor -L/var/lib/managarm-buildenv/build/tools/kernel-gcc/lib/gcc/x86_64-managarm-kernel/13.0.0 -L/var/lib/managarm-buildenv/build/tools/kernel-gcc/lib/gcc/x86_64-managarm-kernel/13.0.0/../../../../x86_64-managarm-kernel/lib/../lib64 -L/var/lib/managarm-buildenv/build/system-root/lib/../lib64 -L/var/lib/managarm-buildenv/build/system-root/usr/lib/../lib64 -L/var/lib/managarm-buildenv/build/tools/kernel-gcc/lib/gcc/x86_64-managarm-kernel/13.0.0/../../../../x86_64-managarm-kernel/lib -L/var/lib/managarm-buildenv/build/system-root/lib -L/var/lib/managarm-buildenv/build/system-root/usr/lib kernel/thor/arch/x86/embed-trampoline.o kernel/thor/thor.p/.._common_libc.cpp.o kernel/thor/thor.p/.._common_font-8x16.cpp.o kernel/thor/thor.p/generic_address-space.cpp.o kernel/thor/thor.p/generic_cancel.cpp.o kernel/thor/thor.p/generic_credentials.cpp.o kernel/thor/thor.p/generic_core.cpp.o kernel/thor/thor.p/generic_debug.cpp.o kernel/thor/thor.p/generic_event.cpp.o kernel/thor/thor.p/generic_fiber.cpp.o kernel/thor/thor.p/generic_gdbserver.cpp.o kernel/thor/thor.p/generic_hel.cpp.o kernel/thor/thor.p/generic_irq.cpp.o kernel/thor/thor.p/generic_io.cpp.o kernel/thor/thor.p/generic_ipc-queue.cpp.o kernel/thor/thor.p/generic_kasan.cpp.o kernel/thor/thor.p/generic_kerncfg.cpp.o kernel/thor/thor.p/generic_kernlet.cpp.o kernel/thor/thor.p/generic_kernel-io.cpp.o kernel/thor/thor.p/generic_kernel-stack.cpp.o kernel/thor/thor.p/generic_main.cpp.o kernel/thor/thor.p/generic_memory-view.cpp.o kernel/thor/thor.p/generic_ostrace.cpp.o kernel/thor/thor.p/generic_physical.cpp.o kernel/thor/thor.p/generic_profile.cpp.o kernel/thor/thor.p/generic_random.cpp.o kernel/thor/thor.p/generic_service.cpp.o kernel/thor/thor.p/generic_schedule.cpp.o kernel/thor/thor.p/generic_stream.cpp.o kernel/thor/thor.p/generic_timer.cpp.o kernel/thor/thor.p/generic_thread.cpp.o kernel/thor/thor.p/generic_servers.cpp.o kernel/thor/thor.p/generic_ubsan.cpp.o kernel/thor/thor.p/generic_universe.cpp.o kernel/thor/thor.p/generic_work-queue.cpp.o kernel/thor/thor.p/system_framebuffer_boot-screen.cpp.o kernel/thor/thor.p/system_framebuffer_fb.cpp.o kernel/thor/thor.p/system_pci_dmalog.cpp.o kernel/thor/thor.p/system_pci_pci_discover.cpp.o kernel/thor/thor.p/system_pci_pci_legacy.cpp.o kernel/thor/thor.p/system_pci_pcie_ecam.cpp.o kernel/thor/thor.p/system_pci_pcie_brcmstb.cpp.o kernel/thor/thor.p/system_legacy-pc_ata.cpp.o kernel/thor/thor.p/arch_x86_early_stubs.S.o kernel/thor/thor.p/arch_x86_entry.S.o kernel/thor/thor.p/arch_x86_stubs.S.o kernel/thor/thor.p/arch_x86_user-access.S.o kernel/thor/thor.p/arch_x86_vmx_stubs.S.o kernel/thor/thor.p/arch_x86_svm_stubs.S.o kernel/thor/thor.p/arch_x86_cpu.cpp.o kernel/thor/thor.p/arch_x86_debug.cpp.o kernel/thor/thor.p/arch_x86_ept.cpp.o kernel/thor/thor.p/arch_x86_hpet.cpp.o kernel/thor/thor.p/arch_x86_ints.cpp.o kernel/thor/thor.p/arch_x86_pic.cpp.o kernel/thor/thor.p/arch_x86_npt.cpp.o kernel/thor/thor.p/arch_x86_paging.cpp.o kernel/thor/thor.p/arch_x86_pmc-amd.cpp.o kernel/thor/thor.p/arch_x86_pmc-intel.cpp.o kernel/thor/thor.p/arch_x86_pci_io.cpp.o kernel/thor/thor.p/arch_x86_rtc.cpp.o kernel/thor/thor.p/arch_x86_svm.cpp.o kernel/thor/thor.p/arch_x86_system.cpp.o kernel/thor/thor.p/arch_x86_vmx.cpp.o kernel/thor/thor.p/system_acpi_glue.cpp.o kernel/thor/thor.p/system_acpi_madt.cpp.o kernel/thor/thor.p/system_acpi_pm-interface.cpp.o kernel/thor/thor.p/system_pci_pci_acpi.cpp.o --as-needed --no-undefined --start-group kernel/thor/liblai.a kernel/thor/libcralgo.a --end-group -z separate-loadable-segments -z max-page-size=0x1000 --image-base 0xFFFFFFFF80000000

[rui314]

I couldn't figure out why it's not working for you. Do you mind if you run the linker with -repro option? That option should create kernel/thor/thor.tar which contains all input object files. Please compress it and share it with me.

[ArsenArsen]

Sure thing, here: thor.repro.tar.gz

[rui314]

I build it from your archive, and here is what I got:

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  PHDR           0x000040 0xffffffff80000040 0xffffffff80000040 0x0001c0 0x0001c0 R   0x8
  INTERP         0x000200 0xffffffff80000200 0xffffffff80000200 0x00001b 0x00001b R   0x1
      [Requesting program interpreter: /lib/x86_64-managarm/ld.so]
  LOAD           0x000000 0xffffffff80000000 0xffffffff80000000 0x020152 0x020152 R   0x1000
  LOAD           0x021000 0xffffffff80021000 0xffffffff80021000 0x2dd8be 0x2dd8be R E 0x1000
  LOAD           0x2ff000 0xffffffff802ff000 0xffffffff802ff000 0x0019f0 0x04a022 RW  0x1000
  GNU_EH_FRAME   0x000eb4 0xffffffff80000eb4 0xffffffff80000eb4 0x0000a4 0x0000a4 R   0x4
  GNU_STACK      0x000000 0x0000000000000000 0x0000000000000000 0x000000 0x000000 RW  0x1
  GNU_RELRO      0x2ff000 0xffffffff802ff000 0xffffffff802ff000 0x000088 0x000088 R   0x1

So it looks like loadable segments are page-aligned. Are you using mold of git head?

[ArsenArsen]

I was on 0763dd5639e26118b95344bbd9f9dfb73f48cd56, so that wasn't it.

The issue was far dumber - I forgot to update the image with a thor rebuilt with this flag. Sorry about that! I'll continue trying to get stuff to boot.

[ArsenArsen]

OK, we got quite far already :)

I've been putting off fixing my broken copy of the userspace, but I'll have to do that next (out of time for the afternoon, though, maybe tonight or tomorrow)

[rui314]

mold now gained new command line options as explained in https://docs.google.com/document/d/1bVRrKsB6iMGmay1odsmS44PyUzPXgQBH83gYGMY6LWM/. With that you can order sections and assign arbitrary addresses to them as you want.

[Qix-]

In my existing linker scripts, I have .multiboot and .multiboot2 sections that are emitted along with the .text section. Using these scripts, grub-file --is-multiboot[2] always exits successfully. I cannot for the life of me figure out how to get mold to do the same.

The current iteration of incantations I've tried is the following. I wish this were better documented as I had to go into the source for mold to figure out the syntax (as it differs significantly from the one defined in the Google doc).

--section-order="EHDR PHDR =0x00100000 .multiboot .multiboot2 .boot %16 TEXT %4096 RODATA %4096 DATA BSS"

No changes except the initial address =0x00100000 changes anything. It's almost like most of the arguments are being ignored. grub-file no longer finds the multiboot structures, and objdump is showing that these sections are well beyond the 0x2000 boundary for multiboot detection;

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00000104  00100000  00100000  00001000  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .eh_frame_hdr 0000000c  00101000  00101000  00002000  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .got          00000008  00102000  00102000  00003000  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  3 .got.plt      0000000c  00102008  00102008  00003008  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  4 .bss          00004000  00102020  00102020  00000000  2**4
                  ALLOC
  5 .comment      0000004f  00000000  00000000  000037ba  2**0
                  CONTENTS, READONLY
  6 .multiboot    0000000c  00000000  00000000  00003810  2**3
                  CONTENTS, READONLY
  7 .multiboot2   00000030  00000000  00000000  00003820  2**3
                  CONTENTS, READONLY

What am I missing here? mold 1.11.0 (46288758a0b6f40a4505f14ea0c95fd06cd10ccb; compatible with GNU ld)

---

Somewhat unrelated, I don't mind linker scripts. I would have loved a little more of a scriptable interface, though - like a lisp of some sort, perhaps. Or at least mold be packaged as a library so that it could be used programmatically. The ecosystem is really lacking a "linker as a library".

[Andy-Python-Programmer]

We would like to implement an alternative mechanism for the linker script language.

I believe that supporting GNU linker scripts would offer advantages by removing the need for multiple build recipes. Moreover, since users can choose their preferred linker, adopting the standard GNU linker script format could facilitate maintenance. Nevertheless, it's important to acknowledge that exploring command line options is a feasible alternative to consider.

[kleinesfilmroellchen]

Using CMake, it appears that mold (1.11) is unable to recognize any linker-script-like arguments. As soon as I can get clang to stop complaining about them, the following happens: mold: fatal: unknown command line option: --start-stop

This is true for --start-stop, --section-order, and --section-align, though the latter appears to have been removed anyways.

Looking at the documentation (not the google document, which is clearly outdated, but the .md file under docs as well as my local manpage), these options also appear nowhere, except --section-align which is referenced by a few other options but never listed itself.

[siltyy]

mold 2.32.1 (cf1c02d097fc57113b7f9fabf1bc15f810b8b27c; compatible with GNU ld)

It's been nearly two years since the kernel/embedded programming support plan was implemented (relevant commits)—why is there still no documentation? --section-order, --start-stop, --section-align, and --physical-image-base are missing from both --help and the manpage, and the already-existing --oformat=binary is included in --help but left out of the manpage. Have I missed an additional source of documentation somewhere?

[rui314]

It's because I'm not sure if these command line options are really the right interface for embedded/kernel programming.

[CorruptedVor]

seL4 linker script https://gist.github.com/CorruptedVor/02db4a720955650308efd8f4ce2dbbfa