Linking xv6-riscv with mold

[ksco]

I'm trying to link mit-pdos/xv6-riscv. Here is the list of features that (I think) mold can't do but is required to build xv6. But if I'm wrong, please correct me.

Basically, we need to add some of the basic functionalities provided by the SECTIONS directive, or something equivalent.

1. ~Specify the order of output sections.~
2. Merge multi-input sections in order into one output section explicitly, with alignment.
3. Get begin/end address of an input section, and assign it to a symbol.

[rui314]

I wonder if you need to specify an in-file order of output sections. As long as they are in some order in the memory, does their physical location in a file matter?

[ksco]

No, it doesn't matter. I just realized this and updated the list.

[rui314]

You may not have to merge input sections to some specific sections. When you write a linker script, you explicitly specify how input sections are mapped to output sections, so it may be more concrete than necessary.

[ksco]

Maybe I should’ve researched more kernels to get some general insights.

But do we want some mold-only directives to achieve this?

[rui314]

If you are saying that adding a mold-only linker script directives, then no, I don't think we want to extend the linker script. We want to instead get rid of linker script completely.

On the other hand, tweaking the OS's code a little bit is totally fine; if we don't do that, we have no choice other than implementing the linker script processor that works 100% compatible with GNU ld.

[ksco]

Sure, then I'll start with xv6, since it's the simplest kernel I know of.

[ksco]

So in xv6-riscv, it needs to put a trampsec page for handling traps at the end of the .text output section. And it needs to know the address of the end of the .text output section, to map the .text section as executable and read-only. Last but not least, it needs to know the address of the end of the kernel, to allocate and free pages above the kernel.

These requirements seem hard to avoid. Do you have any idea how we can implement these features? @rui314

[rui314]

Could you give me a pointer to the actual linker script?

[ksco]

It's in kernel.ld, a simpler (but also works) version is in 035cca95/kernel.ld

[rui314]

For the end symbol, you can just use end symbol which is automatically defined by the linker. See https://github.com/rui314/mold/blob/main/elf/passes.cc#L1563.

For trampsec, can't you use __attribute__((aligned(0x1000)) in the soruce code instead?

[ksco]

Thank you! I think I'm pretty close now.

With this patch: https://github.com/ksco/xv6-riscv/commit/8a55bea8440f421f44fa2c63754a9b5e9184df56, the compiled kernel image seems pretty much the same as the original one.

But still has 2 issues to resolve.

First, when I pass --no-eh-frame-hdr to mold, it segfaulted.

# mold -e=_entry -Ttext 0x80000000 --no-eh-frame-hdr -o kernel/kernel kernel/entry.o kernel/start.o kernel/console.o kernel/printf.o kernel/uart.o kernel/kalloc.o kernel/spinlock.o kernel/string.o kernel/main.o kernel/vm.o kernel/proc.o kernel/swtch.o kernel/trap.o kernel/syscall.o kernel/sysproc.o kernel/bio.o kernel/fs.o kernel/log.o kernel/sleeplock.o kernel/file.o kernel/pipe.o kernel/exec.o kernel/sysfile.o kernel/kernelvec.o kernel/plic.o kernel/virtio_disk.o kernel/trampoline.o
Segmentation fault

I temporarily made the following change to bypass this issue.

// Write to .eh_frame and .eh_frame_hdr.
template <typename E>
void EhFrameSection<E>::copy_buf(Context<E> &ctx) {
+  return;
  u8 *base = ctx.buf + this->shdr.sh_offset;

Second, the address of the trampoline symbol is 0x80007002, which is weird, I'm expecting it to be 0x80007000 like the original image.

    ...

0000000080007002 <trampoline>:
    80007002:   14051073            csrw    sscratch,a0
    80007006:   02000537            lui a0,0x2000

[ksco]

Regarding the second issue, maybe there's a bug in the R_RISCV_ALIGN relaxation?

[rui314]

I'm not sure. What's right at 0x80007000? A nop?

[ksco]

The expected result is the following since trampsec is aligned to 4K.

    ...

0000000080007000 <trampoline>:
    80007000:   14051073            csrw    sscratch,a0
    80007004:   02000537            lui a0,0x2000

[rui314]

Yeah, so, I wondered what's at 0x80007000 in reality.

[ksco]

Yes, it's a nop.

[rui314]

It might be a bug in the R_RISCV_ALIGN processing, but I'm not sure.

[ksco]

Yes, it's a nop.

Correction: It's not a nop, it's 0x0000.

And I think this is a gcc bug.

# cat <<EOF | clang -target riscv64-unknown-linux-gnu -march=rv64gc -mabi=lp64d -fPIC -c -o a.o -xassembler -
.align 2
.section .text.hello
.globl hello
hello:
    li a0, 42
    ret
EOF

# riscv64-linux-gnu-objdump -z -d --reloc a.o 

a.o:     file format elf64-littleriscv

Disassembly of section .text:

0000000000000000 <.text>:
   0:   0001                    nop
                        0: R_RISCV_ALIGN        *ABS*+0x2
   2:   0001                    nop
                        2: R_RISCV_ALIGN        *ABS*+0x2
   4:   0001                    nop
                        4: R_RISCV_ALIGN        *ABS*+0x2

Disassembly of section .text.hello:

0000000000000000 <hello>:
   0:   02a00513                li      a0,42
   4:   8082                    ret

# cat <<EOF | riscv64-linux-gnu-gcc -fPIC -c -o a.o -xassembler -
.align 2
.section .text.hello
.globl hello
hello:
    li a0, 42
    ret
EOF

# riscv64-linux-gnu-objdump -z -d --reloc a.o 

a.o:     file format elf64-littleriscv

Disassembly of section .text:

0000000000000000 <.text>:
   0:   0001                    nop
                        0: R_RISCV_ALIGN        *ABS*+0x2
   2:   0000                    unimp

Disassembly of section .text.hello:

0000000000000000 <hello>:
   0:   02a00513                li      a0,42
   4:   8082                    ret

[ksco]

Good news: With this patch (https://github.com/ksco/xv6-riscv/commit/b53d4242db896461a48a7531fc2ee0622705f2d6), xv6-riscv links correctly with mold and runs fine with qemu and passed all xv6 tests.

[ksco]

But I think this line is a bit tricky: https://github.com/ksco/xv6-riscv/commit/b53d4242db896461a48a7531fc2ee0622705f2d6#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52R31. I put trampoline.o to the end of the object file list, so the trampsec section is in the last of .text output section. But that's not guaranteed by mold, right?

[rui314]

Good news: With this patch (https://github.com/ksco/xv6-riscv/commit/b53d4242db896461a48a7531fc2ee0622705f2d6), xv6-riscv links correctly with mold and runs fine with qemu and passed all xv6 tests.

Great work! It is actually guaranteed that object files are placed in the same order as they are in the command line, so the current code should be fine.

Please report the assembler's issue to GNU binutils.

[ksco]

Please report the assembler's issue to GNU binutils.

I don't have a Sourceware Bugzilla account. They have closed registration and it may take some time to apply for a new account. Or is it convenient for you to do this?

[ksco]

I'll report it once I got an account. Closing issue.

[rui314]

Now that we can build xv6 with mold, do you want to try a more complicated OS kernel?

For xv6, we saw that we didn't really need a linker script. I wonder if it's also true to other OSes.

[ksco]

Yes, I'm looking for the next target. SerenityOS looks good, but it may still be a little complicated? Do you have any recommendations?

[ksco]

Please report the assembler's issue to GNU binutils.

I've reported this issue: https://sourceware.org/bugzilla/show_bug.cgi?id=29557

[rui314]

I didn't have anything specific in my mind. SerenityOS sounds like a good target.