Missing R_PPC_LOCAL24PC and R_PPC_GOT16 for powerpc 32 bit

[Quuxplusone]

Bugzilla Link	PR40888
Status	RESOLVED FIXED
Importance	P enhancement
Reported by	Alfredo Dal'Ava Júnior (alfredo.junior@eldorado.org.br)
Reported on	2019-02-27 08:46:36 -0800
Last modified on	2019-06-25 09:09:26 -0700
Version	unspecified
Hardware	All FreeBSD
CC	chmeeedalf@gmail.com, i@maskray.me, llvm-bugs@lists.llvm.org, luporl@FreeBSD.org, marklmi26-fbsd@yahoo.com, sfertile@ca.ibm.com, smithp352@googlemail.com
Fixed by commit(s)
Attachments	freebsd-lib32-LLD.txt (29201 bytes, text/plain) secure-plt-issue.tar.gz (31118 bytes, application/gzip)
Blocks
Blocked by
See also

I'm trying to build FreeBSD with LLVM (releases/8.x@8c0e32b84d947a00b9689ff5bd1d01447064a71e).

On the 32 bit libraries step of the build I get the error: "can't create dynamic relocation" R_PPC_LOCAL24PC and R_PPC_GOT16.

These are required to build a complete PowerPC64 FreeBSD image with LLVM.

[Quuxplusone]

Hi, can you be more specific about the 32bit libraries step of the build? Which
packages? As you can tell from the number of lines, the PPC32 is pretty much
incomplete. If there isn't much to do, maybe we could hack a bit.

% wc -l Arch/PPC.cpp Arch/PPC64.cpp
   80 Arch/PPC.cpp
  950 Arch/PPC64.cpp
 1030 total

I also believe people have less interests on the 32-bit variant. The newest
SysV ABI Processor Supplement I can find was published in 1995
http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf ...

[Quuxplusone]

(In reply to Fangrui Song from comment #1)

While not explicit, I'm guessing the report is tied to lld use.
Not using lld, and not using llvm's libunwind, last I tried I could
boot a 32-bit pwoerpc FreeBSD head built via FreeBSD's clang. (Dynamic
loading of .ko files was not working so I built in 3 extra modules.)

> I also believe people have less interests on the 32-bit variant. The newest
> SysV ABI Processor Supplement I can find was published in 1995
> http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf ...

powerpc64 FreeBSD by default includes "lib32", which means that it
would be set up to run 32-bit powerpc code (if things were working).
Last I checked for an ELFv1 FreeBSD context, not using llvm's
libunwind, and not using lld, this worked.

[Quuxplusone]

(In reply to Mark Millard from comment #2)

> While not explicit, I'm guessing the report is tied to lld use.
> Not using lld, and not using llvm's libunwind, last I tried I could
> boot a 32-bit pwoerpc FreeBSD head built via FreeBSD's clang. (Dynamic
> loading of .ko files was not working so I built in 3 extra modules.)

I should have been explicit: buildworld was via system-clang (but the
historical FreeBSD binutils toolchain). The buildkernel was via the
historical FreeBSD gcc 4.2.1 and that same binutils.

If I had been more explicit in my thinking, I'd have remembered
the reason why for buildkernel: clang-based got build failures
like . . .

--- agp.ko.full ---
ld: agp.kld(.text+0x3688): R_PPC_PLTREL24 reloc against local symbol
agp.kld: could not read symbols: Bad value
*** [agp.ko.full] Error code 1

(Back in clang 4 days I was able to build a bootable kernel as
I remember.)

[Quuxplusone]

Attached freebsd-lib32-LLD.txt (29201 bytes, text/plain): FreeBSD on PowerPC64 - 32 bit libraries link error

[Quuxplusone]

Hi Alfredo,

I would suggest having a quick look at ELF/ARCH/PPC.cpp as Fangrui pointed out earlier. You will see its really just enough implementation for a very basic hello world type program rather then a fully implemented target. I wouldn't expect it to be able to link a real libc.

The error messages you are getting are likely a red-herring. R_PPC_GOT16 and R_PPC_LOCAL24PC aren't handled in PPC::getRelExpr so they get mapped to R_ABS which allows lld to continue, but handle them in a non-sensical way. I'm not sure why the default case isn't a fatal error but I ran into similar issues when initially implementing the PPC64 target.

Can I ask why you are building the 32-bit libraries on a V2-abi machine? Are 32-bit executables expected to work in a V2-abi environment? My understanding is that it is a 64-bit only environment, but I understand I might have a narrow view of how the abi is used in practice.

[Quuxplusone]

(In reply to sfertile from comment #5)

> Can I ask why you are building the 32-bit libraries on a V2-abi machine? Are
> 32-bit executables expected to work in a V2-abi environment? My
> understanding is that it is a 64-bit only environment, but I understand I
> might have a narrow view of how the abi is used in practice.

The attchment title makes clear the powerpc64 FreeBSD context, so
the following applies.

Quoting comment 2:

powerpc64 FreeBSD by default includes "lib32", which means that it
would be set up to run 32-bit powerpc code (if things were working).

End quote

Also, FreeBSD still has a 32-bit powerpc version (that can
run on some 64-bit hardware as well, such as an old PowerMac
G5).

These are all still big-endian at this point.

I do not know if llvm/clang/lld plans on allowing for clang to
be FreeBSD's system compiler for powerpc64-with-lib32 or not.
Similarly for 32-bit powerpc FreeBSD. Absent such, FreeBSD would
either have to avoid llvm/clang/lld or give up on 32-bit powerpc
(direct or lib32). No such declaration of intent to drop them
have happened yet, despite efforts at tool-chain modernization
support.

[Quuxplusone]

(In reply to sfertile from comment #5, in addition Mark Millard comment #6)

Hi Sean,

Currently, FreeBSD PowerPC64 image supports building and running PowerPC 32-bit applications. The 64-bit sysroot contains 32-bit versions of all system libraries, including libc and the dynamic elf loader. So you can have 32 and 64 bit applications running side by side with their own ABI.

Also, the kernel loaders used on some systems like slof are 32-bit ELF binaries. For this loader specifically, we don't need the 32 bit libraries to compile it. However, ELF binary generated by ld.lld does't work properly (can't load the kernel), and now I understand the reason.

As FreeBSD/PowerPC64 can't drop the 32-bit support at this point, it should continue shipping the GNU ld.bfd (v2.17).

[Quuxplusone]

(In reply to Alfredo Dal'Ava Júnior from comment #7)
> (In reply to sfertile from comment #5, in addition Mark Millard comment #6)
>
> Hi Sean,
>
> Currently, FreeBSD PowerPC64 image supports building and running PowerPC
> 32-bit applications. The 64-bit sysroot contains 32-bit versions of all
> system libraries, including libc and the dynamic elf loader.

Hi Alfredo,

Are you are working on a Power8 or Power9 chip running in little endian mode?
If you are does your build compiler support producing little-endian 32-bit
object files? I didn't think clang could do that but maybe I am mistaken. The
reason I was questioning why you are building the 32-bit binaries is because
I'm not aware if there is a compiler to produce little-endian 32-bit objects.

(In reply to Alfredo Dal'Ava Júnior from comment #7)
> (In reply to sfertile from comment #5, in addition Mark Millard comment #6)
> So you can have 32 and 64 bit applications running side by side with their
> own ABI

My understanding was that the V2 abi was designed as 64-bit *only*. I don't
know if changes were made that would break the loading/running of legacy 32-bit
applications even if they do have the correct endianness, I think this is
something you would have to verify rather then simply take it for granted.
Again I could be mistaken here.

[Quuxplusone]

Ok, I was able to produce a little endian 32-bit object with gcc easy enough -->
gcc -mlittle-endian  -m32 ./other.c -c
objdump --file-headers other.o

other.o:     file format elf32-powerpcle

[Quuxplusone]

(In reply to sfertile from comment #8)
> Are you are working on a Power8 or Power9 chip running in little endian
> mode? If you are does your build compiler support producing little-endian
> 32-bit object files? I didn't think clang could do that but maybe I am
> mistaken. The reason I was questioning why you are building the 32-bit
> binaries is because I'm not aware if there is a compiler to produce
> little-endian 32-bit objects.
>
> (In reply to Alfredo Dal'Ava Júnior from comment #7)
> > (In reply to sfertile from comment #5, in addition Mark Millard comment #6)
> > So you can have 32 and 64 bit applications running side by side with their
> > own ABI
>
> My understanding was that the V2 abi was designed as 64-bit *only*. I don't
> know if changes were made that would break the loading/running of legacy
> 32-bit applications even if they do have the correct endianness, I think
> this is something you would have to verify rather then simply take it for
> granted. Again I could be mistaken here.

FreeBSD uses big endian for powerpc64 and 32-bit powerpc. So far as I
know it does not (yet?) try to support little endian for either.

I'll note that the powerpc64 kernel can be built for one ELF ELF V2 (say
via clang + devel/powerpc64-binutils) and world with ELF V1 (say via gcc
4.2.1 and its toolchain). Or the other way around. The two parts can be
installed and booted as a system.

As for running 32-bit powerpc code on powerpc64, one combination is for
the kernel to be ELF V2 but world to be ELF V1 for both 64-bit and 32-bit
"world material".

I do not claim to know anything about the status of a 64-bit ELF V2
world and the 32-bit ELF V1 as a mixed context. But I've not seen
anything indicating that the 32-bit ABI is to be updated beyond ELF
V1: 32--bit appears to be staying at ELF V1.

[Quuxplusone]

Sfertile,

There's a litte confusion, I'll give thr full context to make it clear.

The FreeBSD ISO image for powerpc64 target are all big endian, there's no
little endian in this environment. This ISO is supposed to run on any powerpc64
hardware like power8, power9 and Mac G5.

Current release FreeBSD 12 uses PPC64 ELFv1 ABI for the kernel and main root
filesystem binaries. It also contains full support for building and running 32
bit binaries in it's original System V ABI [1] (neither ELFv1 or ELFv2, as far
I know) .

This system is built basically with gcc4.2 and binutils ld.bfd 2.17 that's
really old and it got stuck there due to licensing incompatibility.

So, I'm working on the transition to LLVM and as LLD doesn't fully support the
ELFv1 ABI, we are also transitioning to ELFv2 ABI, at same time.

Now we reach a point that LLVM8 + patches [2] are working great building
FreeBSD 13 powerpc64 target, ELFv2 ABI.

But FreeBSD/powerpc64 can't abandon binutils ld.bfd 2.17 yet because it's still
necessary to link powerpc32 system V ABI big endian binaries. As soon LLD
supports this, we will drop binutils ld.bfd completelly.

This bug was meant to report that some reloc may be missing, but as you pointed
out, LLD support for powerpc 32 bit is just in early stages...
Anyway, FreeBSD powerpc* community still has interest in make it usable.

[1] http://refspecs.linux-foundation.org/elf/elfspec_ppc.pdf
[2] https://wiki.freebsd.org/powerpc/llvm-elfv2

[Quuxplusone]

(So in Machine State Register, there is a Sixty-Four-Bit to switch between 32-
bit/64-bit...)
lld has a feature to create reproduce tarballs: --reproduce=/tmp/a.tar . If
cflags/ldflags are cumbersome to change, you may use the environment variable
LLD_REPRODUCE=/tmp/a.tar
I think ppc32 is used by very few applications and we probably don't care about
the performance of ppc32. If we implement none of 1) thunk 2) toc optimization
3) tls , that may be significantly less work :)
Alfredo, can you share reproduce tarballs for some representative applications?
I volunteer to analyze how far we are behind.

[Quuxplusone]

(In reply to Fangrui Song from comment #12)
> Alfredo, can you share reproduce tarballs for some representative
> applications? I volunteer to analyze how far we are behind.

Fangrui, there's a 1MB limit for attachments, so I uploaded the LIB32/LIBC -
LLD reproduce tarball at https://1drv.ms/u/s!AkaL2-RoR7mimZgUica2PKqLFCAiFQ.
Please let me know if you need something else.

The PPC/PPC64 kernel loader is another important software that currently runs
in 32 bit. It's 'static', doesn't depend on LIBC and currently links with LLD
without errors. However the executable doesn't work as expected and cannot load
the kernel.
This one is a bit tricky to debug as there's no OS running at that momment, but
you or someone else mentioned LLD code is assuming 'dubious default behavior'
when something need to be implemented, just to make it go far.

During your analysis, would be nice if you could look for parts where LLD
assumes a default/dubious behavior when something is not implemented and make
it abort/assert/trap instead, so we can see what else is missing for a happy
path that really works.

Thank you for your efforts in making LLVM possible in FreeBSD PowerPC64(and 32)
target!

[Quuxplusone]

It seems FreeBSD rtld only supports -mbss-plt, not -msecure-plt (many Linux gcc
packages are configured with --enable-secure-plt nowadays, i.e. -msecure-plt is
the default). I have a WIP patch to improve the PPC32 port
(https://reviews.llvm.org/D62464) and I am able to run some musl programs with
it. glibc may require some other stuff.

To debug musl: qemu-ppc-static -L /tmp/adelie-linux-rootfs ./a ; (core dumped);
gdb a core
To debug glibc, I am currently using: qemu-system-ppc -hda
debian_wheezy_powerpc_standard.qcow2 -device e1000,netdev=net0 -netdev
user,id=net0,hostfwd=tcp:127.0.0.1:5555-:22 -nographic

For the FreeBSD reproduce tarball, other than the relocation types implemented
in that patch, it needs R_PPC_GOT_TLSGD and R_PPC_GOT_TLSGD16. Do you have a
qemu command to debug FreeBSD rtld?

[Quuxplusone]

Just a comment on here: I've created a WIP for FreeBSD/powerpc (32-bit), adding secure-PLT support, and transitioning it to default for FreeBSD 13. It can be found at https://reviews.freebsd.org/D20598 .

[Quuxplusone]

Attached secure-plt-issue.tar.gz (31118 bytes, application/gzip): secure-plt issue reproduce tarball

[Quuxplusone]

I think we have a problem with clang+secure PLT.
With secure-plt patch[1] applied on FreeBSD, even forcing clang to use it with -
msecure-plt, 9 out of 10 generated .o seems to still be using BSS PLT, so I
could't verify the LLD changes yet.

I've just attached a simple reprodution code.

[1] https://reviews.freebsd.org/D20598

[Quuxplusone]

I think clang -target ppc -msecure-plt -fpic (note, -fpic, not -fPIC) has two
bugs: 1) it doesn't initialize r30 to _GLOBAL_OFFSET_TABLE_ 2) it incorrectly
creates R_PPC_PLTREL24 relocations with 0x8000 r_addend.

ppc-gcc -O2 -msecure-plt -fpic -c a.c

  14:   3f de 00 00     addis   r30,r30,0
                        16: R_PPC_REL16_HA      _GLOBAL_OFFSET_TABLE_+0xa
  18:   90 01 00 14     stw     r0,20(r1)
  1c:   3b de 00 00     addi    r30,r30,0
                        1e: R_PPC_REL16_LO      _GLOBAL_OFFSET_TABLE_+0x12
  20:   48 00 00 01     bl      20 <bar+0x20>
                        20: R_PPC_PLTREL24      foo

clang:

  14:   48 00 00 01     bl      14 <bar+0x14>
                        14: R_PPC_PLTREL24      foo+0x8000

I am investigating the bug. Can you replace -fpic with -fPIC and check if there
is anything else broken?

[Quuxplusone]

(In reply to Fangrui Song from comment #18)
> I am investigating the bug. Can you replace -fpic with -fPIC and check if
> there is anything else broken?

Thank you, for investigating it, Fangrui! I'll try to do this test later today,
before holidays.

In the meanwhile I found that "-msecure-plt" is not enough to force secure-plt,
and this may be the root cause.

In https://reviews.freebsd.org/D20598 jhibbits tries to forces SecurePlt in
method ppc::getPPCReadGOTPtrMode. I tested it and found this is also not enough
to change default PLT mode in clang. I was able to change default value using
the following (draft) change:

diff --git a/contrib/llvm/lib/Target/PowerPC/PPCSubtarget.cpp
b/contrib/llvm/lib/Target/PowerPC/PPCSubtarget.cpp
index 1fdf74549de..9a8f669d894 100644
--- a/contrib/llvm/lib/Target/PowerPC/PPCSubtarget.cpp
+++ b/contrib/llvm/lib/Target/PowerPC/PPCSubtarget.cpp
@@ -138,7 +138,7 @@ void PPCSubtarget::initSubtargetFeatures(StringRef CPU,
StringRef FS) {
   if (isDarwin())
     HasLazyResolverStubs = true;

-  if (TargetTriple.isOSNetBSD() || TargetTriple.isOSOpenBSD())
+  if (TargetTriple.isOSNetBSD() || TargetTriple.isOSOpenBSD() ||
TargetTriple.isOSFreeBSD())
     SecurePlt = true;

   if (HasSPE && IsPPC64)

Apparently there's a mismatch between the SecurePlt variable and
PCReadGOTPtrMode. A correct fix would need to sync these both things.
Also, "-msecure-plt" seems to be useless now, as it doesn't set SecurePlt
variable as far I could understand.

Would you like to handle this in another bug report?

[Quuxplusone]

I haven't checked GOT accesses (it seems jhibbits is working on it) in -msecure-plt -fpic mode.

For the PLT calls, I just created https://reviews.llvm.org/D63563 to fix the bug.

[Quuxplusone]

The clang -target ppc -msecure-plt -fpic bug was fixed by rL364324

Alfredo, you may file new bugs when you find other issues..