Closed ernsteiswuerfel closed 1 year ago
I think this is #602? ld.lld
doesn't support ELFv1, which is the only current option for the kernel with big endian. There is a series for ELFv2 BE on the mailing list but I don't think there is much interest around it.
powerpc64-gentoo-linux-musl does use ELFv2 AFAIK.
From https://git.musl-libc.org/cgit/musl/tree/INSTALL:
* PowerPC
* Compiler toolchain must provide 64-bit long double, not IBM
double-double or IEEE quad
* For dynamic linking, compiler toolchain must be configured for
"secure PLT" variant
* PowerPC64
* Both little and big endian variants are supported
* Compiler toolchain must provide 64-bit long double, not IBM
double-double or IEEE quad
* Compiler toolchain must use the new (ELFv2) ABI regardless of
whether it is for little or big endian
It sounds like you will need to specify CROSS_COMPILE=powerpc64-gentoo-linux-musl
to your make
command then; if CROSS_COMPILE
is not specified, it defaults to a gnu
triple. You might run into other problems after that but they will hopefully be easy to debug.
Thanks for the hint! But when I build with
# LLVM=1 LLVM_IAS=1 CROSS_COMPILE=powerpc64-gentoo-linux-musl make
I still get the same error.
Looks like there are some updated patches from Nicholas Piggin to enable ELFv2 ABI on ppc64 kernel builds.
I'll report back here as soon I tested them. Probably on weekend. π
Meanwhile the new (experimental) config option PPC64_BIG_ENDIAN_ELF_ABI_V2
appeared in kernel v6.2-rc. After enabling it I get these settings in the kernel .config:
CONFIG_PPC64_ELF_ABI_V2=y
CONFIG_CC_HAS_ELFV2=y
CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y
Built v6.2-rc3 and now v6.2-rc4 with these setting and the resulting runs pretty well and without any hickups! So far so good.
Next I tried do to a Clang kernel build with # LLVM=1 LLVM_IAS=1 CROSS_COMPILE=powerpc64-gentoo-linux-musl make
but it does not pick up the PPC64_BIG_ENDIAN_ELF_ABI_V2
option, I only get:
CONFIG_PPC64_ELF_ABI_V1=y
CONFIG_CC_HAS_ELFV2=y
in the .config afterwards. Building such a kernel gives me the same ld.lld: error: undefined symbol: .early_setup
I originally reported.
It looks like the relevant config option is specifically checking for BFD linker and if negative won't appear in menuconfig:
Symbol: PPC64_BIG_ENDIAN_ELF_ABI_V2 [=n]
Type : bool
Defined at arch/powerpc/Kconfig:602
Prompt: Build big-endian kernel using ELF ABI V2 (EXPERIMENTAL)
Depends on: PPC64 [=y] && CPU_BIG_ENDIAN [=y] && CC_HAS_ELFV2 [=y] && LD_IS_BFD [=y] && LD_VERSION [=23900]>=22400
Location:
-> Kernel options
-> Build big-endian kernel using ELF ABI V2 (EXPERIMENTAL) (PPC64_BIG_ENDIAN_ELF_ABI_V2 [=n])
I bypassed the check by deleting line 606 depends on LD_IS_BFD && LD_VERSION >= 22400 in arch/powerpc/Kconfig. Afterwards config option PPC64_BIG_ENDIAN_ELF_ABI_V2
does appear and CONFIG_PPC64_ELF_ABI_V2=y
, CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y
are set accordingly.
Then after some building I am one error furtner. π
# LLVM=1 LLVM_IAS=1 CROSS_COMPILE=powerpc64-gentoo-linux-musl make
CALL scripts/checksyscalls.sh
BOOTAS arch/powerpc/boot/crt0.o
error: unknown target ABI 'elfv2'
make[1]: *** [arch/powerpc/boot/Makefile:232: arch/powerpc/boot/crt0.o] Error 1
make: *** [arch/powerpc/Makefile:247: zImage] Error 2
@ernsteiswuerfel Thanks for taking a look! I have a WIP series here, which allows me to build your original configuration linked in the report with CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y
successfully with LLVM 15.0.7 and LLVM 16.0.0 (recent tip of tree). Consider taking it for a spin to see if everything works properly.
@nathanchance Thanks for your patches! Just gave them a try, built a kernel for my PowerMac G5 and it booted out of the box without issues! π₯³ Have to do some stress testing the next few days and see if I can break a thing and try the same with a clang-built kernel on my Talos II too. π
Binary size is smaller (gcc zImage 7.2 MiB vs. clang zImage 6.6 MiB), performance seems on par.
Only thing I did notice so far is an Altivec performance regression in the in-kernel xor benchmark:
GCC 12
xor: measuring software checksum speed
8regs : 6353 MB/sec
8regs_prefetch : 6053 MB/sec
32regs : 6541 MB/sec
32regs_prefetch : 5984 MB/sec
altivec : 9199 MB/sec
xor: using function: altivec (9199 MB/sec)
CLANG 15
xor: measuring software checksum speed
8regs : 6340 MB/sec
8regs_prefetch : 5133 MB/sec
32regs : 6330 MB/sec
32regs_prefetch : 4989 MB/sec
altivec : 892 MB/sec
xor: using function: 8regs (6340 MB/sec)
However not all Altivec code is affected, the in-kernel raid6 benchmark looks ok with clang:
GCC 12
raid6: altivecx8 gen() 6231 MB/s
raid6: altivecx4 gen() 6385 MB/s
raid6: altivecx2 gen() 5903 MB/s
raid6: altivecx1 gen() 4082 MB/s
raid6: int64x8 gen() 2798 MB/s
raid6: int64x4 gen() 3063 MB/s
raid6: int64x2 gen() 2532 MB/s
raid6: int64x1 gen() 1743 MB/s
raid6: using algorithm altivecx4 gen() 6385 MB/s
raid6: using intx1 recovery algorithm
CLANG 15
raid6: altivecx8 gen() 6121 MB/s
raid6: altivecx4 gen() 6329 MB/s
raid6: altivecx2 gen() 6424 MB/s
raid6: altivecx1 gen() 3862 MB/s
raid6: int64x8 gen() 2601 MB/s
raid6: int64x4 gen() 2910 MB/s
raid6: int64x2 gen() 2803 MB/s
raid6: int64x1 gen() 1543 MB/s
raid6: using algorithm altivecx2 gen() 6424 MB/s
raid6: using intx1 recovery algorithm
Just gave them a try, built a kernel for my PowerMac G5 and it booted out of the box without issues! π₯³
That is great to hear! Good to know all the work they did was toolchain agnostic :)
Binary size is smaller (gcc zImage 7.2 MiB vs. clang zImage 6.6 MiB), performance seems on par.
Only thing I did notice so far is an Altivec performance regression in the in-kernel xor benchmark:
GCC 12 xor: measuring software checksum speed 8regs : 6353 MB/sec 8regs_prefetch : 6053 MB/sec 32regs : 6541 MB/sec 32regs_prefetch : 5984 MB/sec altivec : 9199 MB/sec xor: using function: altivec (9199 MB/sec) CLANG 15 xor: measuring software checksum speed 8regs : 6340 MB/sec 8regs_prefetch : 5133 MB/sec 32regs : 6330 MB/sec 32regs_prefetch : 4989 MB/sec altivec : 892 MB/sec xor: using function: 8regs (6340 MB/sec)
However not all Altivec code is affected, the in-kernel raid6 benchmark looks ok with clang:
GCC 12 raid6: altivecx8 gen() 6231 MB/s raid6: altivecx4 gen() 6385 MB/s raid6: altivecx2 gen() 5903 MB/s raid6: altivecx1 gen() 4082 MB/s raid6: int64x8 gen() 2798 MB/s raid6: int64x4 gen() 3063 MB/s raid6: int64x2 gen() 2532 MB/s raid6: int64x1 gen() 1743 MB/s raid6: using algorithm altivecx4 gen() 6385 MB/s raid6: using intx1 recovery algorithm CLANG 15 raid6: altivecx8 gen() 6121 MB/s raid6: altivecx4 gen() 6329 MB/s raid6: altivecx2 gen() 6424 MB/s raid6: altivecx1 gen() 3862 MB/s raid6: int64x8 gen() 2601 MB/s raid6: int64x4 gen() 2910 MB/s raid6: int64x2 gen() 2803 MB/s raid6: int64x1 gen() 1543 MB/s raid6: using algorithm altivecx2 gen() 6424 MB/s raid6: using intx1 recovery algorithm
Huh, interesting... I wonder if that is related to #1713?
Huh, interesting... I wonder if that is related to #1713?
Yes, just like that! Updated this issue with my ppc64 kernel disassembly.
Clang-built kernel works fine on my Talos II too! Binary size is a bit bigger (gcc zImage 31 MiB vs. clang zImage 34 MiB), performance seems a little bit slower. Same Altivec performance regression as on the G5.
Your patches did their job very well, thanks! Would be nice if they could go mainline for 6.2. π
Edit: dmesg attached dmesg_62-rc6_p9-clang.txt dmesg_62-rc6_p9-gcc.txt
Thanks for testing! Would you like me to add a Tested-by
tag for you?
Would be nice if they could go mainline for 6.2.
Unfortunately, it is extremely unlikely those patches would be seen as 6.2 material, as they are enablement patches, rather than fixes.
I will try to get them sent out tomorrow or Wednesday so that they have a chance of making 6.3.
Thanks for testing! Would you like me to add a
Tested-by
tag for you?
Sure, you can add me.
Unfortunately, it is extremely unlikely those patches would be seen as 6.2 material, as they are enablement patches, rather than fixes.
I will try to get them sent out tomorrow or Wednesday so that they have a chance of making 6.3.
Ok, thanks!
My thought was that CONFIG_PPC64_ELF_ABI_V2=y
is new in 6,2 and if clang enablement would land in 6.2 as well you would have the option on BE to choose between GCC and CLANG from the start. But if CLANG enablement lands in the kernel later on it's not a big deal.
Sorry for that delay, I have now sent https://lore.kernel.org/20230118-ppc64-elfv2-llvm-v1-0-b9e2ec9da11d@kernel.org/.
Slightly more stable link I think:
https://git.kernel.org/powerpc/l/a11334d8327b3fd7987cbfb38e956a44c722d88f
I plan to wire up allmodconfig
+ CONFIG_PPC64_BIG_ENDIAN_ELF_ABI_V2=y
in CI today.
Hey Nathan, thanks for that ELFv2 for BE patch. What I'm wondering is, you say ld.lld doesn't support ELFv1, but what happens if we try to build it with ELFv1? Shouldn't we force ELFv2 for all LLVM builds, and prohibit BE kernels being built with LLD < 15 if there is a concern about correctness?
what happens if we try to build it with ELFv1
The build will fail to link: https://github.com/ClangBuiltLinux/linux/issues/602
Shouldn't we force ELFv2 for all LLVM builds, and prohibit BE kernels being built with LLD < 15 if there is a concern about correctness?
This would not be a bad idea, we already force ld.bfd
with big endian ELFv1 configurations in our CI, so that would not be that big of a deal to prohibit ld.lld
builds with big endian ELFv1 for ld.lld
< 15 and require ELFv2 for ld.lld
>= 15.
My series is now in mainline: https://git.kernel.org/torvalds/l/a11334d8327b3fd7987cbfb38e956a44c722d88f
@nathanchance , are there configs we should test ppc64 with LLD in CI?
@nathanchance , are there configs we should test ppc64 with LLD in CI?
I already turned on allmodconfig
plus ELFv2 with the whole LLVM stack in -next, I'll push the change for mainline later today.
Kernel + modules build fine but I get an error at the linking stage:
System is powerpc64-gentoo-linux-musl, build is done natively on my PowerMac G5 11,2 or Talos II. Toolchain used was llvm/clang/lld-15.0.3.
Same kernel builds fine with gcc-12.2, binutils-2.39. Kernel .config attached config_61-rc5_g5.txt