search

ClangBuiltLinux / linux

Linux kernel source tree
Other
241 stars 14 forks source link

clang 15 fails ppc64 BE kernel build - ld.lld: error: undefined symbol: .early_setup (kernel 6.1-rc5, powerpc64-gentoo-linux-musl) #1761

Closed ernsteiswuerfel closed 1 year ago

ernsteiswuerfel commented 1 year ago

Kernel + modules build fine but I get an error at the linking stage:

 # LLVM=1 LLVM_IAS=1 make
[...]
  CALL    scripts/checksyscalls.sh
  UPD     include/generated/utsversion.h
  CC      init/version-timestamp.o
  LD      .tmp_vmlinux.kallsyms1
ld.lld: error: undefined symbol: .early_setup
>>> referenced by head_64.S:973
>>> (/usr/src/linux-stable/arch/powerpc/kernel/head_64.S:973)
>>>               arch/powerpc/kernel/head_64.o:(.ref.text+0xBE) in archive
>>>               vmlinux.a
>>> referenced by head_64.S:973
>>> (/usr/src/linux-stable/arch/powerpc/kernel/head_64.S:973)
>>>               arch/powerpc/kernel/head_64.o:(.ref.text+0xC2) in archive
>>>               vmlinux.a
>>> did you mean: early_setup
>>> defined in: vmlinux.a(arch/powerpc/kernel/setup_64.o)
make[1]: *** [scripts/Makefile.vmlinux:34: vmlinux] Error 1
make: *** [Makefile:1236: vmlinux] Error 2

System is powerpc64-gentoo-linux-musl, build is done natively on my PowerMac G5 11,2 or Talos II. Toolchain used was llvm/clang/lld-15.0.3.

Same kernel builds fine with gcc-12.2, binutils-2.39. Kernel .config attached config_61-rc5_g5.txt

nathanchance commented 1 year ago

I think this is #602? ld.lld doesn't support ELFv1, which is the only current option for the kernel with big endian. There is a series for ELFv2 BE on the mailing list but I don't think there is much interest around it.

ernsteiswuerfel commented 1 year ago

powerpc64-gentoo-linux-musl does use ELFv2 AFAIK.

From https://git.musl-libc.org/cgit/musl/tree/INSTALL:

* PowerPC
    * Compiler toolchain must provide 64-bit long double, not IBM
      double-double or IEEE quad
    * For dynamic linking, compiler toolchain must be configured for
      "secure PLT" variant

* PowerPC64
    * Both little and big endian variants are supported
    * Compiler toolchain must provide 64-bit long double, not IBM
      double-double or IEEE quad
    * Compiler toolchain must use the new (ELFv2) ABI regardless of
      whether it is for little or big endian
nathanchance commented 1 year ago

It sounds like you will need to specify CROSS_COMPILE=powerpc64-gentoo-linux-musl to your make command then; if CROSS_COMPILE is not specified, it defaults to a gnu triple. You might run into other problems after that but they will hopefully be easy to debug.

ernsteiswuerfel commented 1 year ago

Thanks for the hint! But when I build with # LLVM=1 LLVM_IAS=1 CROSS_COMPILE=powerpc64-gentoo-linux-musl make I still get the same error.

ernsteiswuerfel commented 1 year ago

Looks like there are some updated patches from Nicholas Piggin to enable ELFv2 ABI on ppc64 kernel builds.

I'll report back here as soon I tested them. Probably on weekend. 😉

ernsteiswuerfel commented 1 year ago

Meanwhile the new (experimental) config option PPC64_BIG_ENDIAN_ELF_ABI_V2 appeared in kernel v6.2-rc. After enabling it I get these settings in the kernel .config:

CONFIG_PPC64_ELF_ABI_V2=y
CONFIG_CC_HAS_ELFV2=y
CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y

Built v6.2-rc3 and now v6.2-rc4 with these setting and the resulting runs pretty well and without any hickups! So far so good.

Next I tried do to a Clang kernel build with # LLVM=1 LLVM_IAS=1 CROSS_COMPILE=powerpc64-gentoo-linux-musl make but it does not pick up the PPC64_BIG_ENDIAN_ELF_ABI_V2 option, I only get:

CONFIG_PPC64_ELF_ABI_V1=y
CONFIG_CC_HAS_ELFV2=y

in the .config afterwards. Building such a kernel gives me the same ld.lld: error: undefined symbol: .early_setup I originally reported.

It looks like the relevant config option is specifically checking for BFD linker and if negative won't appear in menuconfig:

Symbol: PPC64_BIG_ENDIAN_ELF_ABI_V2 [=n]
Type  : bool
Defined at arch/powerpc/Kconfig:602
  Prompt: Build big-endian kernel using ELF ABI V2 (EXPERIMENTAL)
  Depends on: PPC64 [=y] && CPU_BIG_ENDIAN [=y] && CC_HAS_ELFV2 [=y] && LD_IS_BFD [=y] && LD_VERSION [=23900]>=22400
  Location:
    -> Kernel options
      -> Build big-endian kernel using ELF ABI V2 (EXPERIMENTAL) (PPC64_BIG_ENDIAN_ELF_ABI_V2 [=n])

I bypassed the check by deleting line 606 depends on LD_IS_BFD && LD_VERSION >= 22400 in arch/powerpc/Kconfig. Afterwards config option PPC64_BIG_ENDIAN_ELF_ABI_V2 does appear and CONFIG_PPC64_ELF_ABI_V2=y, CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y are set accordingly.

Then after some building I am one error furtner. 😉 

 # LLVM=1 LLVM_IAS=1 CROSS_COMPILE=powerpc64-gentoo-linux-musl make
  CALL    scripts/checksyscalls.sh
  BOOTAS  arch/powerpc/boot/crt0.o
error: unknown target ABI 'elfv2'
make[1]: *** [arch/powerpc/boot/Makefile:232: arch/powerpc/boot/crt0.o] Error 1
make: *** [arch/powerpc/Makefile:247: zImage] Error 2
nathanchance commented 1 year ago

@ernsteiswuerfel Thanks for taking a look! I have a WIP series here, which allows me to build your original configuration linked in the report with CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y successfully with LLVM 15.0.7 and LLVM 16.0.0 (recent tip of tree). Consider taking it for a spin to see if everything works properly.

ernsteiswuerfel commented 1 year ago

@nathanchance Thanks for your patches! Just gave them a try, built a kernel for my PowerMac G5 and it booted out of the box without issues! 🥳 Have to do some stress testing the next few days and see if I can break a thing and try the same with a clang-built kernel on my Talos II too. 😉

Binary size is smaller (gcc zImage 7.2 MiB vs. clang zImage 6.6 MiB), performance seems on par.

Only thing I did notice so far is an Altivec performance regression in the in-kernel xor benchmark:

   GCC 12
xor: measuring software checksum speed
   8regs           :  6353 MB/sec
   8regs_prefetch  :  6053 MB/sec
   32regs          :  6541 MB/sec
   32regs_prefetch :  5984 MB/sec
   altivec         :  9199 MB/sec
xor: using function: altivec (9199 MB/sec)

   CLANG 15
xor: measuring software checksum speed
   8regs           :  6340 MB/sec
   8regs_prefetch  :  5133 MB/sec
   32regs          :  6330 MB/sec
   32regs_prefetch :  4989 MB/sec
   altivec         :   892 MB/sec
xor: using function: 8regs (6340 MB/sec)

However not all Altivec code is affected, the in-kernel raid6 benchmark looks ok with clang:

   GCC 12
raid6: altivecx8 gen()  6231 MB/s
raid6: altivecx4 gen()  6385 MB/s
raid6: altivecx2 gen()  5903 MB/s
raid6: altivecx1 gen()  4082 MB/s
raid6: int64x8  gen()  2798 MB/s
raid6: int64x4  gen()  3063 MB/s
raid6: int64x2  gen()  2532 MB/s
raid6: int64x1  gen()  1743 MB/s
raid6: using algorithm altivecx4 gen() 6385 MB/s
raid6: using intx1 recovery algorithm

   CLANG 15
raid6: altivecx8 gen()  6121 MB/s
raid6: altivecx4 gen()  6329 MB/s
raid6: altivecx2 gen()  6424 MB/s
raid6: altivecx1 gen()  3862 MB/s
raid6: int64x8  gen()  2601 MB/s
raid6: int64x4  gen()  2910 MB/s
raid6: int64x2  gen()  2803 MB/s
raid6: int64x1  gen()  1543 MB/s
raid6: using algorithm altivecx2 gen() 6424 MB/s
raid6: using intx1 recovery algorithm

dmesg_62-rc4_g5.txt

nathanchance commented 1 year ago

Just gave them a try, built a kernel for my PowerMac G5 and it booted out of the box without issues! 🥳

That is great to hear! Good to know all the work they did was toolchain agnostic :)

Binary size is smaller (gcc zImage 7.2 MiB vs. clang zImage 6.6 MiB), performance seems on par.

Only thing I did notice so far is an Altivec performance regression in the in-kernel xor benchmark:

   GCC 12
xor: measuring software checksum speed
   8regs           :  6353 MB/sec
   8regs_prefetch  :  6053 MB/sec
   32regs          :  6541 MB/sec
   32regs_prefetch :  5984 MB/sec
   altivec         :  9199 MB/sec
xor: using function: altivec (9199 MB/sec)

   CLANG 15
xor: measuring software checksum speed
   8regs           :  6340 MB/sec
   8regs_prefetch  :  5133 MB/sec
   32regs          :  6330 MB/sec
   32regs_prefetch :  4989 MB/sec
   altivec         :   892 MB/sec
xor: using function: 8regs (6340 MB/sec)

However not all Altivec code is affected, the in-kernel raid6 benchmark looks ok with clang:

   GCC 12
raid6: altivecx8 gen()  6231 MB/s
raid6: altivecx4 gen()  6385 MB/s
raid6: altivecx2 gen()  5903 MB/s
raid6: altivecx1 gen()  4082 MB/s
raid6: int64x8  gen()  2798 MB/s
raid6: int64x4  gen()  3063 MB/s
raid6: int64x2  gen()  2532 MB/s
raid6: int64x1  gen()  1743 MB/s
raid6: using algorithm altivecx4 gen() 6385 MB/s
raid6: using intx1 recovery algorithm

   CLANG 15
raid6: altivecx8 gen()  6121 MB/s
raid6: altivecx4 gen()  6329 MB/s
raid6: altivecx2 gen()  6424 MB/s
raid6: altivecx1 gen()  3862 MB/s
raid6: int64x8  gen()  2601 MB/s
raid6: int64x4  gen()  2910 MB/s
raid6: int64x2  gen()  2803 MB/s
raid6: int64x1  gen()  1543 MB/s
raid6: using algorithm altivecx2 gen() 6424 MB/s
raid6: using intx1 recovery algorithm

dmesg_62-rc4_g5.txt

Huh, interesting... I wonder if that is related to #1713?

ernsteiswuerfel commented 1 year ago

Huh, interesting... I wonder if that is related to #1713?

Yes, just like that! Updated this issue with my ppc64 kernel disassembly.

ernsteiswuerfel commented 1 year ago

Clang-built kernel works fine on my Talos II too! Binary size is a bit bigger (gcc zImage 31 MiB vs. clang zImage 34 MiB), performance seems a little bit slower. Same Altivec performance regression as on the G5.

Your patches did their job very well, thanks! Would be nice if they could go mainline for 6.2. 😃

Edit: dmesg attached dmesg_62-rc6_p9-clang.txt dmesg_62-rc6_p9-gcc.txt

nathanchance commented 1 year ago

Thanks for testing! Would you like me to add a Tested-by tag for you?

Would be nice if they could go mainline for 6.2.

Unfortunately, it is extremely unlikely those patches would be seen as 6.2 material, as they are enablement patches, rather than fixes.

I will try to get them sent out tomorrow or Wednesday so that they have a chance of making 6.3.

ernsteiswuerfel commented 1 year ago

Thanks for testing! Would you like me to add a Tested-by tag for you?

Sure, you can add me.

Unfortunately, it is extremely unlikely those patches would be seen as 6.2 material, as they are enablement patches, rather than fixes.

I will try to get them sent out tomorrow or Wednesday so that they have a chance of making 6.3.

Ok, thanks!

My thought was that CONFIG_PPC64_ELF_ABI_V2=y is new in 6,2 and if clang enablement would land in 6.2 as well you would have the option on BE to choose between GCC and CLANG from the start. But if CLANG enablement lands in the kernel later on it's not a big deal.

nathanchance commented 1 year ago

Sorry for that delay, I have now sent https://lore.kernel.org/20230118-ppc64-elfv2-llvm-v1-0-b9e2ec9da11d@kernel.org/.

nickdesaulniers commented 1 year ago

accepted: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/log/?h=next

nathanchance commented 1 year ago

Slightly more stable link I think:

https://git.kernel.org/powerpc/l/a11334d8327b3fd7987cbfb38e956a44c722d88f

I plan to wire up allmodconfig + CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y in CI today.

npiggin commented 1 year ago

Hey Nathan, thanks for that ELFv2 for BE patch. What I'm wondering is, you say ld.lld doesn't support ELFv1, but what happens if we try to build it with ELFv1? Shouldn't we force ELFv2 for all LLVM builds, and prohibit BE kernels being built with LLD < 15 if there is a concern about correctness?

nathanchance commented 1 year ago

what happens if we try to build it with ELFv1

The build will fail to link: https://github.com/ClangBuiltLinux/linux/issues/602

Shouldn't we force ELFv2 for all LLVM builds, and prohibit BE kernels being built with LLD < 15 if there is a concern about correctness?

This would not be a bad idea, we already force ld.bfd with big endian ELFv1 configurations in our CI, so that would not be that big of a deal to prohibit ld.lld builds with big endian ELFv1 for ld.lld < 15 and require ELFv2 for ld.lld >= 15.

nathanchance commented 1 year ago

My series is now in mainline: https://git.kernel.org/torvalds/l/a11334d8327b3fd7987cbfb38e956a44c722d88f

nickdesaulniers commented 1 year ago

@nathanchance , are there configs we should test ppc64 with LLD in CI?

nathanchance commented 1 year ago

@nathanchance , are there configs we should test ppc64 with LLD in CI?

I already turned on allmodconfig plus ELFv2 with the whole LLVM stack in -next, I'll push the change for mainline later today.