ld.lld build failures for powerpc starting with next-20200128

[nathanchance]

NOTE: Because of #774, clang needs to be configured to use ld.lld by default (-DCLANG_DEFAULT_LINKER=lld) to see this. Otherwise, adding -fuse-ld=lld to ccflags-y in arch/powerpc/kernel/vdso32/Makefile should work.

$ make -j$(nproc) -s ARCH=powerpc CC=clang CROSS_COMPILE=powerpc-linux-gnu- LD=ld.lld O=out.ppc32 distclean allnoconfig all
ld.lld: error: relocation R_PPC_REL16_LO cannot be used against symbol __kernel_datapage_offset; recompile with -fPIC
>>> defined in arch/powerpc/kernel/vdso32/datapage.o
>>> referenced by arch/powerpc/kernel/vdso32/gettimeofday.o:(__kernel_gettimeofday)

ld.lld: error: relocation R_PPC_REL16_LO cannot be used against symbol __kernel_datapage_offset; recompile with -fPIC
>>> defined in arch/powerpc/kernel/vdso32/datapage.o
>>> referenced by arch/powerpc/kernel/vdso32/gettimeofday.o:(__kernel_clock_gettime)

ld.lld: error: relocation R_PPC_REL16_LO cannot be used against symbol __kernel_datapage_offset; recompile with -fPIC
>>> defined in arch/powerpc/kernel/vdso32/datapage.o
>>> referenced by arch/powerpc/kernel/vdso32/gettimeofday.o:(__kernel_clock_getres)

ld.lld: error: relocation R_PPC_REL16_LO cannot be used against symbol __kernel_datapage_offset; recompile with -fPIC
>>> defined in arch/powerpc/kernel/vdso32/datapage.o
>>> referenced by arch/powerpc/kernel/vdso32/gettimeofday.o:(__kernel_time)

ld.lld: error: relocation R_PPC_REL16_LO cannot be used against symbol __kernel_datapage_offset; recompile with -fPIC
>>> defined in arch/powerpc/kernel/vdso32/datapage.o
>>> referenced by arch/powerpc/kernel/vdso32/cacheflush.o:(__kernel_sync_dicache)
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
../arch/powerpc/kernel/vdso32/Makefile:40: recipe for target 'arch/powerpc/kernel/vdso32/vdso32.so.dbg' failed
make[5]: *** [arch/powerpc/kernel/vdso32/vdso32.so.dbg] Error 1
../scripts/Makefile.build:503: recipe for target 'arch/powerpc/kernel/vdso32' failed
make[4]: *** [arch/powerpc/kernel/vdso32] Error 2
make[4]: *** Waiting for unfinished jobs....
../scripts/Makefile.build:503: recipe for target 'arch/powerpc/kernel' failed
make[3]: *** [arch/powerpc/kernel] Error 2
make[3]: *** Waiting for unfinished jobs....
...

Caused by https://git.kernel.org/powerpc/c/ec0895f08f99515194e9fcfe1338becf6f759d38.

[nickdesaulniers]

hmm, I wonder if -fPIC should be added to ccflags-y in arch/powerpc/kernel/vdso32/Makefile?

Either gcc doesn't generate R_PPC_REL16_LO relocations, or LLD is wrong about that error message. It looks like a relative relocation, so I'd think adding -fPIC is the way to go. cc @MaskRay

[nathanchance]

It does look like space and x86 add -fpic to the CFLAGS for their 32-bit vDSOs so maybe there is precedent there. At the same time, I wonder if there is something deeper going on in ld.lld.

[MaskRay]

What is next-20200128? master is good. Should I add git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git as another remote?

edit:

git remote add linux-next git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
git fetch linux-next
git checkout next-20200204

% git diff -U1
diff --git i/arch/powerpc/kernel/vdso32/Makefile w/arch/powerpc/kernel/vdso32/Makefile
index 06f54d947057..30eb22b3f36b 100644
--- i/arch/powerpc/kernel/vdso32/Makefile
+++ w/arch/powerpc/kernel/vdso32/Makefile
@@ -29,3 +29,3 @@ UBSAN_SANITIZE := n
 ccflags-y := -shared -fno-common -fno-builtin -nostdlib \
-       -Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both
+       -Wl,-soname=linux-vdso32.so.1 -Wl,--hash-style=both -fuse-ld=lld
 asflags-y := -D__VDSO32__ -s

% make -j30 ARCH=powerpc CC=/tmp/ReleaseA/bin/clang CROSS_COMPILE=powerpc-linux-gnu- LD=/tmp/ReleaseA/bin/ld.lld O=/tmp/out/ppc32 all
ld.lld: error: relocation R_PPC_REL16_LO cannot be used against symbol __kernel_datapage_offset; recompile with -fPIC                                 
>>> defined in arch/powerpc/kernel/vdso32/datapage.o                                                                                                  
>>> referenced by arch/powerpc/kernel/vdso32/gettimeofday.o:(__kernel_gettimeofday)
...

Delete arch/powerpc/kernel/vdso32/vdso32.so.dbg, rerun make all V=1. I get the linker command line: clang ... -shared ... -o arch/powerpc/kernel/vdso32/vdso32.so.dbg

% readelf -s /tmp/out/ppc32/arch/powerpc/kernel/vdso32/vdso32.so.dbg | grep __kernel_datapage_offset
     4: 000005f4     0 NOTYPE  GLOBAL DEFAULT    8 __kernel_datapage_offset@@LINUX_2.6.15
    13: 000005f4     0 NOTYPE  GLOBAL DEFAULT    8 __kernel_datapage_offset

The symbol is preemptible in a -shared link. + addi \ptr, \ptr, (__kernel_datapage_offset - (.-4))@l in https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id=ec0895f08f99515194e9fcfe1338becf6f759d38 introduced an incorrect PC-relative relocation. R_PPC_REL16_LO is a bit special in GNU ld. GNU ld seems to allow it referencing a preemptible symbol, though it is not the case on other architectures.

I can send a patch. How should I start? :)

[nathanchance]

@MaskRay

http://nickdesaulniers.github.io/blog/2017/05/16/submitting-your-first-patch-to-the-linux-kernel-and-responding-to-feedback/

https://www.kernel.org/doc/html/latest/process/submitting-patches.html

1. Create the change, test it, and commit it with your explanation and signed-off-by tag.

2. Run git format-patch -1 to generate the patch file.

3. Run ./scripts/checkpatch.pl <patch_file> to check the patch and make sure there are no errors.

4. Run ./scripts/get_maintainer.pl <patch_file> to get the maintainers of the file.

Sample output:

$ ./scripts/get_maintainer.pl arch/powerpc/kernel/vdso32/
Benjamin Herrenschmidt <benh@kernel.crashing.org> (supporter:LINUX FOR POWERPC (32-BIT AND 64-BIT))
Paul Mackerras <paulus@samba.org> (supporter:LINUX FOR POWERPC (32-BIT AND 64-BIT))
Michael Ellerman <mpe@ellerman.id.au> (supporter:LINUX FOR POWERPC (32-BIT AND 64-BIT),commit_signer:12/16=75%)
Christophe Leroy <christophe.leroy@c-s.fr> (commit_signer:11/16=69%,authored:9/16=56%)
Thomas Gleixner <tglx@linutronix.de> (commit_signer:2/16=12%,authored:2/16=12%)
Greg Kroah-Hartman <gregkh@linuxfoundation.org> (commit_signer:2/16=12%)
Allison Randal <allison@lohutok.net> (commit_signer:2/16=12%)
Arnd Bergmann <arnd@arndb.de> (authored:2/16=12%)
Nick Desaulniers <ndesaulniers@google.com> (authored:1/16=6%)
Vincenzo Frascino <vincenzo.frascino@arm.com> (authored:1/16=6%)
linuxppc-dev@lists.ozlabs.org (open list:LINUX FOR POWERPC (32-BIT AND 64-BIT))
linux-kernel@vger.kernel.org (open list)

5. Use git send-email to send the patch. For something touching this, I would recommend something like this (maintainers, person who introduced the error with the PPC, main Linux kernel, and our mailing list on CC):

$ git send-email --to="Benjamin Herrenschmidt <benh@kernel.crashing.org>" \
                           --to="Paul Mackerras <paulus@samba.org>" \
                           --to="Michael Ellerman <mpe@ellerman.id.au>" \
                           --to="Christophe Leroy <christophe.leroy@c-s.fr>" \
                           --cc="linuxppc-dev@lists.ozlabs.org" \
                           --cc="linux-kernel@vger.kernel.org" \
                           --cc="clang-built-linux@googlegroups.com" \
                           <patch_file>

Feel free to ask me any questions :)

[MaskRay]

Thanks! I'll try that tomorrow.

In the meantime, I filed https://sourceware.org/bugzilla/show_bug.cgi?id=25500 . This is a missing diagnostic from the powerpc32/powerpc64 ports of GNU ld.

[nathanchance]

https://lore.kernel.org/lkml/20200205005054.k72fuikf6rwrgfe4@google.com/

[nickdesaulniers]

needs to be chased

[nickdesaulniers]

It sounds like __get_datapage may have been removed from the kernel?

[nathanchance]

Does not look like it. Christophe's patch series that he mentioned no longer includes a patch that removes __get_datapage so Fangrui's patch needs to be chased.

[nathanchance]

Still relevant but marking low priority for now because this cannot be reproduced without -DCLANG_DEFAULT_LINKER=lld passed to cmake at LLVM build time or #774 resolved.

[nathanchance]

Actually, this was resolved with https://git.kernel.org/linus/49bf59fd0371b1053a17021f27605f43071584ee so closing.