MIPS: vmlinuz does not boot if LD=ld.lld

[pcercuei]

When compiling for a Ingenic JZ4770 MIPS board (make gcw0_defconfig), using CC=clang and LLVM tools, the generated vmlinux file will boot fine. However, the vmlinuz file won't boot unless the linker used for the self-extracting archive is forced (in arch/mips/boot/compressed/Makefile) to GNU's linker.

[nickdesaulniers]

Thanks for the report. I don't see any specific compression routine specified in the config. I'd be curious which one of these gets selected by that config? IIRC, there was a fairly recent kernel patch about misaligned accesses during decompression on MIPS...

[pcercuei]

The CONFIG_KERNEL_GZIP is selected by default. But this issue seems unrelated to the compression scheme. It is not a problem of misaligned access either: I wrote the patch you are referring to and I have it applied.

Sometimes (not always), when compressed with ZSTD, the self-extracting executable will throw errors saying that the "ZSTD is corrupt", or "Out of memory while allocating ZSTD_DCtx" which never happens when linked with GNU's ld.

I think the problem lies somewhere in arch/mips/boot/compressed/ld.script.

See the diff when running mipsel-linux-objdump -x vmlinuz (with - being GNU LD, + being LLVM LD): sections.txt

I am not certain what is going on, but there are some strange things in there; for instance, .appended_dtb changes from ALLOC to CONTENTS, ALLOC, LOAD, DATA...

[arndb]

Are you able to reproduce the problem with a kernel running in qemu?

[pcercuei]

@arndb let me try that and get back to you.

[pcercuei]

@arndb qemu seems to boot the vmlinuz correctly. Unfortunately I don't think qemu supports booting a vmlinuz.bin.

[pcercuei]

Still happens in 5.15-rc1 :(

[nathanchance]

I hit #1546 when building vmlinuz for gcw0_defconfig. Once I patch that up with the second diff in that issue, I can link.

$ mipsel-linux-gnu-objdump -x vmlinuz.bfd

vmlinuz.bfd:     file format elf32-tradlittlemips
vmlinuz.bfd
architecture: mips:isa32r2, flags 0x00000102:
EXEC_P, D_PAGED
start address 0x81000000

Program Header:
0x70000003 off    0x00013938 vaddr 0x81003938 paddr 0x81003938 align 2**3
         filesz 0x00000018 memsz 0x00000018 flags r--
    LOAD off    0x00010000 vaddr 0x81000000 paddr 0x81000000 align 2**16
         filesz 0x0042c8c0 memsz 0x0092e8d0 flags rwx
private flags = 70001001: [abi=O32] [mips32r2] [not 32bitmode] [noreorder]

MIPS ABI Flags Version: 0

ISA: MIPS32r2
GPR size: 32
CPR1 size: 0
CPR2 size: 0
FP ABI: Soft float
ISA Extension: None
ASEs:
        None
FLAGS 1: 00000001
FLAGS 2: 00000000

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00003934  81000000  81000000  00010000  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .MIPS.abiflags 00000018  81003938  81003938  00013938  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE
  2 .rodata.str1.1 00000164  81003950  81003950  00013950  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .data         00428e00  81003ac0  81003ac0  00013ac0  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  4 .appended_dtb 00100000  8142c8c0  8142c8c0  0043c8c0  2**0
                  ALLOC
  5 .bss          00402010  8152c8c0  8152c8c0  0043c8c0  2**4
                  ALLOC
SYMBOL TABLE:
no symbols

$ mipsel-linux-gnu-objdump -x vmlinuz.lld

vmlinuz.lld:     file format elf32-tradlittlemips
vmlinuz.lld
architecture: mips:isa32r2, flags 0x00000102:
EXEC_P, D_PAGED
start address 0x81000000

Program Header:
    LOAD off    0x00010000 vaddr 0x81000000 paddr 0x81000000 align 2**16
         filesz 0x0052c8b4 memsz 0x0092e8d0 flags rwx
0x70000003 off    0x0043c738 vaddr 0x8142c738 paddr 0x8142c738 align 2**3
         filesz 0x00000018 memsz 0x00000018 flags r--
private flags = 70001001: [abi=O32] [mips32r2] [not 32bitmode] [noreorder]

MIPS ABI Flags Version: 0

ISA: MIPS32r2
GPR size: 32
CPR1 size: 0
CPR2 size: 0
FP ABI: Soft float
ISA Extension: None
ASEs:
        None
FLAGS 1: 00000001
FLAGS 2: 00000000

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00003934  81000000  81000000  00010000  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .data         00428df0  81003940  81003940  00013940  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  2 .got          00000008  8142c730  8142c730  0043c730  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  3 .MIPS.abiflags 00000018  8142c738  8142c738  0043c738  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE
  4 .rodata.str1.1 00000164  8142c750  8142c750  0043c750  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .appended_dtb 00100000  8142c8b4  8142c8b4  0043c8b4  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  6 .bss          00402010  8152c8c0  8152c8c0  0053c8b4  2**4
                  ALLOC
SYMBOL TABLE:
no symbols

diff --git a/objdump.bfd b/objdump.lld
index 546b6bd..3feda33 100644
--- a/objdump.bfd
+++ b/objdump.lld
@@ -1,15 +1,15 @@

-vmlinuz.bfd:     file format elf32-tradlittlemips
-vmlinuz.bfd
+vmlinuz.lld:     file format elf32-tradlittlemips
+vmlinuz.lld
 architecture: mips:isa32r2, flags 0x00000102:
 EXEC_P, D_PAGED
 start address 0x81000000

 Program Header:
-0x70000003 off    0x00013938 vaddr 0x81003938 paddr 0x81003938 align 2**3
-         filesz 0x00000018 memsz 0x00000018 flags r--
     LOAD off    0x00010000 vaddr 0x81000000 paddr 0x81000000 align 2**16
-         filesz 0x0042c8c0 memsz 0x0092e8d0 flags rwx
+         filesz 0x0052c8b4 memsz 0x0092e8d0 flags rwx
+0x70000003 off    0x0043c738 vaddr 0x8142c738 paddr 0x8142c738 align 2**3
+         filesz 0x00000018 memsz 0x00000018 flags r--
 private flags = 70001001: [abi=O32] [mips32r2] [not 32bitmode] [noreorder]

 MIPS ABI Flags Version: 0
@@ -29,15 +29,17 @@ Sections:
 Idx Name          Size      VMA       LMA       File off  Algn
   0 .text         00003934  81000000  81000000  00010000  2**4
                   CONTENTS, ALLOC, LOAD, READONLY, CODE
-  1 .MIPS.abiflags 00000018  81003938  81003938  00013938  2**3
+  1 .data         00428df0  81003940  81003940  00013940  2**4
+                  CONTENTS, ALLOC, LOAD, DATA
+  2 .got          00000008  8142c730  8142c730  0043c730  2**4
+                  CONTENTS, ALLOC, LOAD, DATA
+  3 .MIPS.abiflags 00000018  8142c738  8142c738  0043c738  2**3
                   CONTENTS, ALLOC, LOAD, READONLY, DATA, LINK_ONCE_SAME_SIZE
-  2 .rodata.str1.1 00000164  81003950  81003950  00013950  2**0
+  4 .rodata.str1.1 00000164  8142c750  8142c750  0043c750  2**0
                   CONTENTS, ALLOC, LOAD, READONLY, DATA
-  3 .data         00428e00  81003ac0  81003ac0  00013ac0  2**4
+  5 .appended_dtb 00100000  8142c8b4  8142c8b4  0043c8b4  2**0
                   CONTENTS, ALLOC, LOAD, DATA
-  4 .appended_dtb 00100000  8142c8c0  8142c8c0  0043c8c0  2**0
-                  ALLOC
-  5 .bss          00402010  8152c8c0  8152c8c0  0043c8c0  2**4
+  6 .bss          00402010  8152c8c0  8152c8c0  0053c8b4  2**4
                   ALLOC
 SYMBOL TABLE:
 no symbols

@pcercuei's section diff has a bunch of function sections, was CONFIG_LD_DEAD_CODE_DATA_ELIMINATION turned on for those builds? If so, perhaps that is causing issues and something like below might remedy it?

diff --git a/arch/mips/boot/compressed/ld.script b/arch/mips/boot/compressed/ld.script
index 0ebb667274d6..561ad4cf37a6 100644
--- a/arch/mips/boot/compressed/ld.script
+++ b/arch/mips/boot/compressed/ld.script
@@ -16,14 +16,14 @@ SECTIONS
        /* Text and read-only data */
        /* . = VMLINUZ_LOAD_ADDRESS; */
        .text : {
-               *(.text)
-               *(.rodata)
+               *(.text .text.*)
+               *(.rodata .rodata.*)
        }: text
        /* End of text section */

        /* Writable data */
        .data : {
-               *(.data)
+               *(.data .data.*)
                /* Put the compressed image here */
                __image_begin = .;
                *(.image)
@@ -43,7 +43,7 @@ SECTIONS

        /* BSS */
        .bss : {
-               *(.bss)
+               *(.bss .bss.*)
        }
        . = ALIGN(16);
        _end = .;

[pcercuei]

@nathanchance I had CONFIG_LD_DEAD_CODE_DATA_ELIMINATION enabled, that's correct. However, neither disabling this option nor applying your patch seem to fix the problem.

[nathanchance]

Okay, good to know, one less variable to deal with.

When you say that it does not boot, do you get any output on the console like a crash or does it just hang? Does MIPS have an earlycon option?

[pcercuei]

There is an earlycon option, and there's an option to enable logging in the decompressor itself. Here's what I get over UART:

zimage at:     810133B0 813F73E6
Uncompressing Linux at load address 80100000
Now, booting the kernel...

Then no output at all, which means that the kernel crashes very early.

Some progress though: if I embed the DTB into the kernel (gcw0.dtb is built-in if CONFIG_MIPS_NO_APPENDED_DTB) then it does boot. In my config (the one which does not work) I do use CONFIG_MIPS_RAW_APPENDED_DTB instead and append the .dtb to the vmlinuz.bin file. I guess this is what is broken; if the kernel cannot find the device tree it won't print anything on UART.

[nathanchance]

@pcercuei Based on that UART output, it seems like the if statement in decompress_kernel() is getting skipped? Shouldn't we see Copy device tree to address ...?

[nickdesaulniers]

diff --git a/objdump.bfd b/objdump.lld

Why is .data and .MIPS.abiflags reordered between linkers? Are they not placed explicitly in a linker script? Should LD_ORPHAN_WARN warn about that?

[pcercuei]

@nathanchance We definitely should, yes. If I print &__appended_dtb I get 0x813F72D8, which does sound very wrong. It should be 0x814F72D8 (as my vmlinuz.bin file is 0x4F72D8 bytes).

[nathanchance]

Why is .data and .MIPS.abiflags reordered between linkers? Are they not placed explicitly in a linker script? Should LD_ORPHAN_WARN warn about that?

CONFIG_LD_ORPHAN_WARN is only added to LDFLAGS_vmlinux, not KBUILD_LDFLAGS, which is what vmlinuz uses. Perhaps it should be?

Adding the flag manually to the zld command, I see:

ld.lld: warning: arch/mips/boot/compressed/head.o:(.debug_frame) is being placed in '.debug_frame'
ld.lld: warning: arch/mips/boot/compressed/decompress.o:(.mdebug.abi32) is being placed in '.mdebug.abi32'
ld.lld: warning: arch/mips/boot/compressed/decompress.o:(.rodata.str1.1) is being placed in '.rodata.str1.1'
ld.lld: warning: arch/mips/boot/compressed/string.o:(.mdebug.abi32) is being placed in '.mdebug.abi32'
ld.lld: warning: arch/mips/boot/compressed/bswapsi.o:(.mdebug.abi32) is being placed in '.mdebug.abi32'
ld.lld: warning: arch/mips/boot/compressed/piggy.o:(.mdebug.abi32) is being placed in '.mdebug.abi32'
ld.lld: warning: arch/mips/boot/compressed/head.o:(COMMON) is being placed in '.bss'
ld.lld: warning: arch/mips/boot/compressed/head.o:(COMMON) is being placed in '.bss'
ld.lld: warning: <internal>:(.MIPS.abiflags) is being placed in '.MIPS.abiflags'
ld.lld: warning: <internal>:(.got) is being placed in '.got'
ld.lld: warning: <internal>:(.symtab) is being placed in '.symtab'
ld.lld: warning: <internal>:(.shstrtab) is being placed in '.shstrtab'
ld.lld: warning: <internal>:(.strtab) is being placed in '.strtab'

None of those looks particularly damning.

One interesting thing is that arch/mips/boot/vmlinux.lds.S has __appended_dtb outside of a section, whereas arch/mips/boot/compressed/vmlinux.S has it within .appended_dtb. Could that be a problem?

[nickdesaulniers]

Perhaps it should be?

Unless configs intentionally reorder these, which I highly doubt, I suspect "yes."

Could that be a problem?

Which section? One that's being reordered differently?

[nathanchance]

Which section? One that's being reordered differently?

.appended_dtb, it is being reordered and having its section flags changed, as Paul notes in his second comment.

[nickdesaulniers]

Then that's probably a problem; I'll bet that __appended_dtb is a linker script defined symbol meant to denote the start of .append_dtb?

[nathanchance]

diff --git a/arch/mips/boot/compressed/ld.script b/arch/mips/boot/compressed/ld.script
index 0ebb667274d6..a8b0c3ed514d 100644
--- a/arch/mips/boot/compressed/ld.script
+++ b/arch/mips/boot/compressed/ld.script
@@ -32,11 +32,9 @@ SECTIONS
                . = ALIGN(16);
        }

-       .appended_dtb : {
-               __appended_dtb = .;
-               /* leave space for appended DTB */
-               . += 0x100000;
-       }
+       __appended_dtb = .;
+       /* leave space for appended DTB */
+       . += 0x100000;

        _edata = .;
        /* End of data section */

I wonder if that changes anything? I don't see the .appended_dtb section used anywhere in arch/mips/compressed/boot/...

[pcercuei]

So we actually don't care at all about a .appended_dtb section; we just need the __appended_dtb symbol to point exactly to the byte after the end of the vmlinuz.bin file.