search

Quuxplusone / LLVMBugzillaTest

0 stars 0 forks source link

[META] Using LLD as FreeBSD's system linker #23213

Closed Quuxplusone closed 3 years ago

Quuxplusone commented 9 years ago
Bugzilla Link PR23214
Status RESOLVED FIXED
Importance P normal
Reported by emaste@freebsd.org
Reported on 2015-04-13 10:48:45 -0700
Last modified on 2020-11-04 12:16:23 -0800
Version unspecified
Hardware PC FreeBSD
CC alfredo.junior@eldorado.org.br, anton@korobeynikov.info, ditaliano@apple.com, erik@cederstrand.dk, glaubitz@physik.fu-berlin.de, grimar@accesssoftek.com, jsweval@arxan.com, kevin.bowling@kev009.com, lffpires@ruabrasil.org, lists@eitanadler.com, llvm-bugs@lists.llvm.org, marklmi26-fbsd@yahoo.com, mpysw@vip.163.com, rafael@espindo.la, rdivacky@freebsd.org, rengolin@gmail.com, ruiu@google.com, rwindz0@gmail.com, shawn.webb@hardenedbsd.org, willdtz@gmail.com
Fixed by commit(s)
Attachments loader.zip (1298516 bytes, application/x-zip-compressed)
Blocks
Blocked by PR20976, PR22906, PR23024, PR23035, PR23231, PR23232, PR23233, PR25188, PR25528, PR25587, PR25790, PR25798, PR26676, PR26693, PR26699, PR26705, PR26712, PR26729, PR26730, PR26731, PR26732, PR26737, PR26813, PR26814, PR26818, PR26878, PR26968, PR27016, PR27228, PR27406, PR27407, PR27647, PR27893, PR28349, PR28357, PR28359, PR28414, PR28688, PR28689, PR28690, PR28720, PR28976, PR28999, PR29093, PR29115, PR30218, PR30221, PR30222, PR30237, PR30268, PR30269, PR30282, PR30311, PR30312, PR30406, PR30415, PR30891, PR31057, PR31148, PR31149, PR31159, PR31196, PR31213, PR31221, PR31277, PR31295, PR31495, PR31582, PR31619, PR31716, PR31762, PR32504, PR33422, PR33423, PR33482, PR33766, PR33820, PR33821, PR34164, PR34311, PR35570, PR35720, PR35751, PR35788, PR36009, PR36017
See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229050, PR30267, PR25176

This is a meta-bug to track dependencies needed for LLD to be functional as FreeBSD's system linker.

Quuxplusone commented 7 years ago

A brief high-level status update:

For the set of armv7l, arm64, i386, x86_64 LLD can now link the FreeBSD kernel and modules. On all but i386 the kernel and modules have been tested to boot and confirm basic functionality. i386 kernel has been booted. Kernel modules link but are untested.

LLD 4.0.0 is in the FreeBSD base system source tree and is:

1) The default linker used to build the FreeBSD/arm64 base system (kernel+modules, userland) and installed as /usr/bin/ld, in HEAD and will be in the upcoming 11.1 release.

2) Usable for the FreeBSD/amd64 (x86-64) base system, with investigation on failures in the 27000 pieces of software in the ports tree ongoing. I am using LLD as /usr/bin/ld on my daily driver FreeBSD desktop and laptop.

LLD HEAD has been imported into the FreeBSD clang500-import branch and is:

3) Usable as the linker for the FreeBSD/armv6 kernel+modules. Note that FreeBSD does not yet have a separate target for armv7; TARGET_ARCH=armv6 is what we use for the majority of our 32-bit arm devices. Linking libc fails due to PR 31159; testing has been done on an lld-linked kernel+modules with a ld.bfd-linked userland.

4) Usable as the linker for the FreeBSD/i386 kernel+modules. lld-linked kernel boots. Modules build but are untested. Linking the base system succeeds, but applications all segfault with assertion errors from jemalloc. No further investigation or testing has yet been done.

Quuxplusone commented 7 years ago 
Two recent regressions affecting FreeBSD/arm64 are described in:
https://reviews.llvm.org/rL307100
https://reviews.llvm.org/rL307364
Quuxplusone commented 6 years ago

FYI as of r327783 (https://reviews.freebsd.org/rS327783) FreeBSD is using lld as the bootstrap linker, to link the kernel and base system userland libraries and binaries on amd64. We're still using ld.bfd as /usr/bin/ld and all of the ports collection will thus be linked with bfd today, but I expect to change this in another couple of weeks.

I expect we will then make the same series of changes for i386, and armv7 after. (FreeBSD/arm64 has been using lld as the bootstrap linker and /usr/bin/ld for some time already.)

After that we'll move on to MIPS and PowerPC.

Big thanks to everyone who's worked on lld, and in particular to those who have handled the bug reports that are listed as dependencies of this.

(I expect I'll keep this PR open to track any final issues until we're using lld as the bootstrap linker and /usr/bin/ld on all Tier-1 architectures.)

Quuxplusone commented 6 years ago 
(In reply to emaste from comment #53)

> . . .
>
> After that we'll move on to MIPS and PowerPC.
>
> . . .

Just an FYI about the status of clang for
powerpc and powerpc64 for FreeBSD, taking
specific example points (not everything). . .

clang for powerpc and powerpc64 does not correctly
implement __builtin_eh_return [llvm bugzilla 28644].
As a consequence if clang is used to build the code
that uses __builtin_eh_return then throwing a C++
exception causes the program to crash, even for
something as simple as:

#include <exception>

int main(void)
{
    try { throw std::exception(); }
    catch (std::exception& e) {}
    return 0;
}

(As I understand __builtin_eh_return is basically
unimplemented as far as its consequences go in what
is generated.)

Also, for powerpc64, the switch to clang 5.0.1 changed
to use .rela.plt and R_PPC64_JMP_SLOT in kernel module
builds but the kernel does not handle dynamically
loading such, crashing the kernel during the load.
(The older .rela.dyn with R_PPC64_RELATIVE and
R_PPC64_ADDR64 usage was working.) I do not know
enough to know which side should change for this
issue and so have not submitted it to llvm so far.
Quuxplusone commented 6 years ago 
(In reply to Mark Millard from comment #54)
>
> Also, for powerpc64, the switch to clang 5.0.1 changed
> to use .rela.plt and R_PPC64_JMP_SLOT in kernel module
> builds but the kernel does not handle dynamically
> loading such, crashing the kernel during the load.
> (The older .rela.dyn with R_PPC64_RELATIVE and
> R_PPC64_ADDR64 usage was working.) I do not know
> enough to know which side should change for this
> issue and so have not submitted it to llvm so far.

I wonder if it can be related with fact that PPC64 incorrectly
works with PLT sections atm (D41613 addresses this).
Quuxplusone commented 6 years ago 
(In reply to George Rimar from comment #55)
> (In reply to Mark Millard from comment #54)
> >
> > Also, for powerpc64, the switch to clang 5.0.1 changed
> > to use .rela.plt and R_PPC64_JMP_SLOT in kernel module
> > builds but the kernel does not handle dynamically
> > loading such, crashing the kernel during the load.
> > (The older .rela.dyn with R_PPC64_RELATIVE and
> > R_PPC64_ADDR64 usage was working.) I do not know
> > enough to know which side should change for this
> > issue and so have not submitted it to llvm so far.
>
> I wonder if it can be related with fact that PPC64 incorrectly
> works with PLT sections atm (D41613 addresses this).

I finally updated to a more modern FreeBSD vintage
that is based on clang 6 as the system compiler:

# clang --version
FreeBSD clang version 6.0.0 (branches/release_60 325330) (based on LLVM 6.0.0)
Target: x86_64-unknown-freebsd12.0
Thread model: posix
InstalledDir: /usr/bin

It still gets the .rela.plt and R_PPC64_JMP_SLOT use that
5.0.1 did. (Cross compile based buildworld buildkernel .)

I've not checked if branches/release_60 325330 should
have D41613 or not: I'm just reporting FreeBSD's status
for the issue for now.
Quuxplusone commented 6 years ago 
(In reply to George Rimar from comment #55)
> (In reply to Mark Millard from comment #54)
> >
> > Also, for powerpc64, the switch to clang 5.0.1 changed
> > to use .rela.plt and R_PPC64_JMP_SLOT in kernel module
> > builds but the kernel does not handle dynamically
> > loading such, crashing the kernel during the load.
> > (The older .rela.dyn with R_PPC64_RELATIVE and
> > R_PPC64_ADDR64 usage was working.) I do not know
> > enough to know which side should change for this
> > issue and so have not submitted it to llvm so far.
>
> I wonder if it can be related with fact that PPC64 incorrectly
> works with PLT sections atm (D41613 addresses this).

I looked at D41613 a little and it appears to just
be for lld. The above type of problem happens even
if devel/powerpc64-binutils (or equivalent) is used
with clang 5.0.1 or clang 6. (So far, this is the
normal way to experiment with clang targeting
powerpc64, at least in my experiments.)

I'm not clear on just were the responsibilities are
for this, both for FreeBSD vs. llvm and for clang
vs. lld vs. both on the llvm side of things. Primarily
I'm just reporting the overall mismatch, not resolving
how and where it should be handled.
Quuxplusone commented 6 years ago 
I added a see-also back to the FreeBSD PR tracking outstanding issues in
switching to LLD as the system linker --
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229050

Very brief per-arch summary, for FreeBSD-current (i.e., SVN head):

- arm64 and amd64 use lld for the bootstrap and system linker (i.e., done)
- i386 was blocked on LLVM PR 37735 which is now fixed and cherry-picked into
FreeBSD -- testing is underway before making the switch
- armv7 is blocked on LLVM PR 36009
- armv6, arm (v4/v5) blocked on use of movt/movw, extended branch encoding, blx
wh- mips successful proof of concept demonstrated
- ppc/power userland linking fails with various relocation erros
- sparc64 not investigated
- riscv64 waiting on upstream support
Quuxplusone commented 5 years ago 
Up-to-date FreeBSD-current status:

- lld is the bootstrap and system linker for arm64, armv7, amd64 (aka x86_64),
and i386
- armv6 has not yet been switched but lld 7 is now in the FreeBSD base system
and we should be able to enable it there
- arm (v4/v5) support is expected to be retired and will probably not switch to
lld before then
- mips is still at "proof of concept demonstrated"; we will likely be able to
switch mips64 to Clang + lld soon
- no progress on ppc/power, sparc64, riscv64
Quuxplusone commented 4 years ago

lld is (or is functional as, and soon will be) the bootstrap and system linker for all FreeBSD supported architectures except 32-bit powerpc and sparc64

Quuxplusone commented 4 years ago 
(In reply to Mark Millard from comment #54)
> (In reply to emaste from comment #53)
>
> > . . .
> >
> > After that we'll move on to MIPS and PowerPC.
> >
> > . . .
>
> Just an FYI about the status of clang for
> powerpc and powerpc64 for FreeBSD, taking
> specific example points (not everything). . .
>
> clang for powerpc and powerpc64 does not correctly
> implement __builtin_eh_return [llvm bugzilla 28644].
> As a consequence if clang is used to build the code
> that uses __builtin_eh_return then throwing a C++
> exception causes the program to crash, even for
> something as simple as:
>
> #include <exception>
>
> int main(void)
> {
>     try { throw std::exception(); }
>     catch (std::exception& e) {}
>     return 0;
> }
>
> (As I understand __builtin_eh_return is basically
> unimplemented as far as its consequences go in what
> is generated.)
>
> Also, . . .

FreeBSD has implicitly sidestepped the __builtin_eh_return
issue by using llvm's libunwind, which does not depend on
__builtin_eh_return causing the exception handling scratch
registers r3-r6 to have cfi information.

It is still the case that a libunwind library that is designed
to require that cfi information can not be usefully built and
used via clang: __builtin_eh_return does not lead to the
cfi content required (Itanium ABI variant for powerpc64 or
32-bit powerpc).

For FreeBSD, that should only matter if someone tried to build
FreeBSD via WITHOUT_LLVM_LIBUNWIND= .
Quuxplusone commented 3 years ago

lld is now used as the system linker for all archs supported by FreeBSD

One depends-on issue was not closed (PR30267). It has been worked around in FreeBSD and has been moved to the see-also list here.

Thank you to everyone who helped make this possible!