search

Quuxplusone / LLVMBugzillaTest

0 stars 0 forks source link

[META] Using LLD as FreeBSD's system linker #23213

Closed Quuxplusone closed 3 years ago

Quuxplusone commented 9 years ago
Bugzilla Link PR23214
Status RESOLVED FIXED
Importance P normal
Reported by emaste@freebsd.org
Reported on 2015-04-13 10:48:45 -0700
Last modified on 2020-11-04 12:16:23 -0800
Version unspecified
Hardware PC FreeBSD
CC alfredo.junior@eldorado.org.br, anton@korobeynikov.info, ditaliano@apple.com, erik@cederstrand.dk, glaubitz@physik.fu-berlin.de, grimar@accesssoftek.com, jsweval@arxan.com, kevin.bowling@kev009.com, lffpires@ruabrasil.org, lists@eitanadler.com, llvm-bugs@lists.llvm.org, marklmi26-fbsd@yahoo.com, mpysw@vip.163.com, rafael@espindo.la, rdivacky@freebsd.org, rengolin@gmail.com, ruiu@google.com, rwindz0@gmail.com, shawn.webb@hardenedbsd.org, willdtz@gmail.com
Fixed by commit(s)
Attachments loader.zip (1298516 bytes, application/x-zip-compressed)
Blocks
Blocked by PR20976, PR22906, PR23024, PR23035, PR23231, PR23232, PR23233, PR25188, PR25528, PR25587, PR25790, PR25798, PR26676, PR26693, PR26699, PR26705, PR26712, PR26729, PR26730, PR26731, PR26732, PR26737, PR26813, PR26814, PR26818, PR26878, PR26968, PR27016, PR27228, PR27406, PR27407, PR27647, PR27893, PR28349, PR28357, PR28359, PR28414, PR28688, PR28689, PR28690, PR28720, PR28976, PR28999, PR29093, PR29115, PR30218, PR30221, PR30222, PR30237, PR30268, PR30269, PR30282, PR30311, PR30312, PR30406, PR30415, PR30891, PR31057, PR31148, PR31149, PR31159, PR31196, PR31213, PR31221, PR31277, PR31295, PR31495, PR31582, PR31619, PR31716, PR31762, PR32504, PR33422, PR33423, PR33482, PR33766, PR33820, PR33821, PR34164, PR34311, PR35570, PR35720, PR35751, PR35788, PR36009, PR36017
See also https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229050, PR30267, PR25176

This is a meta-bug to track dependencies needed for LLD to be functional as FreeBSD's system linker.

Quuxplusone commented 9 years ago

Do you have a list of missing features for this bug?

Quuxplusone commented 9 years ago 
There are the depends-on PRs:

* We need __start_<section> and __stop_<section> as we use linker sets in a few
programs
* -r relocatable link

I just closed PR23035 as the change was reapplied with a fixed test case, but
have found a new issue relating to dependency handling (no PR yet):

link main.o -l<static> -l<shared>, where libstatic.a depends on libshared.a.
LLD reports an undefined symbol from libstatic.a. GNU ld accepts this. Arguably
the link order ought to be changed around in our build, but other software will
likely encounter this too.

For the actual case in FreeBSD the static library was NTP - it builds a
libntp.a that depends on libedit, and a binary linked with -ledit -lntp.

I also observed one "error: unsupported emulation 'elf_i386_fbsd'."

PRs for these two and other issues we find will be added as dependencies of
this one.
Quuxplusone commented 9 years ago

We also need linker script support, although I don't have a list of the specific linker script features we rely on just yet.

Quuxplusone commented 9 years ago 
Unknown arguments during world (userland) build:

warning: ignoring unknown argument: --version-script=Version.map
warning: ignoring unknown argument: -x
warning: ignoring unknown argument: --fatal-warnings
warning: ignoring unknown argument: --warn-shared-textrel

From FreeBSD arm64 kernel link:

--- kernel.debug ---
linking kernel.debug
warning: ignoring unknown argument: -Bdynamic
warning: ignoring unknown argument: --no-warn-mismatch
warning: ignoring unknown argument: --warn-common
warning: ignoring unknown argument: --dynamic-linker
warning: ignoring unknown argument: -X
SymbolTable: error while merging __bss_start
LLVM ERROR: duplicate symbol error
Quuxplusone commented 9 years ago

Note that this meta PR was created based on the first version of LLD ELF support, and needs to be revisited for elf2.

Quuxplusone commented 8 years ago 
Trying with ELF2 turned up the issues below; we're not yet at the point in
ELF2's development where it makes sense to track individual PRs for them.

* PT_INTERP and ABI notes must be in the first page

ELF2 currently rounds up to a page and stores the data for PT_INTERP
and the NT_FREEBSD_ABI_TAG and NT_FREEBSD_NOINIT_TAG ELF notes there.
These have to be in the first page on FreeBSD.

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flg    Align
  INTERP         0x00000000000010cc 0x00000000000110cc 0x00000000000110cc
                 0x0000000000000015 0x0000000000000015  R      0x1
      [Requesting program interpreter: /libexec/ld-elf.so.1]

* Undefined sym in shared library provided by regular object

This is the FreeBSD __progname issue fixed in r218259 for the original
ELF linker.

* Options used in the FreeBSD base system

These are options that need to be ignored or implemented (excluding
those already ignored by ELF2)

--enable-new-dtags
--fatal-warnings and --no-fatal-warnings
--warn-shared-textrel
--output-filetype
--version-script
-Bsymbolic
-r
-znodelete
--no-warn-mismatch
--warn-common
-T

* Linker script support

The FreeBSD base system linker scripts use (at least) the following set:
ALIGN ENTRY GROUP KEEP OUTPUT_ARCH OUTPUT_FORMAT SEARCH_DIR SECTIONS PROVIDE

(http://lists.llvm.org/pipermail/llvm-dev/2015-October/091072.html)
Quuxplusone commented 8 years ago

--enable/disable-new-dtags is r249428

Quuxplusone commented 8 years ago

What kind of expressions are used in SECTIONS linker script directive?

Quuxplusone commented 8 years ago 
> What kind of expressions are used in SECTIONS linker script directive?

Examples can be found here: https://svnweb.freebsd.org/base/head/sys/conf/
For amd64: https://svnweb.freebsd.org/base/head/sys/conf/ldscript.amd64?view=log

Some examples:

  kernphys = CONSTANT (MAXPAGESIZE);
  . = kernbase + kernphys + SIZEOF_HEADERS;
  .interp         : AT (kernphys + SIZEOF_HEADERS) { *(.interp) }
  .hash           : { *(.hash) }
  .rel.text       : { *(.rel.text .rel.text.* .rel.gnu.linkonce.t.*) }

  .init           :
  {
    KEEP (*(.init))
  } =0x90909090

  PROVIDE (__etext = .);

  .eh_frame       : ONLY_IF_RO { KEEP (*(.eh_frame)) }

  . = ALIGN (CONSTANT (MAXPAGESIZE)) - ((CONSTANT (MAXPAGESIZE) - .) & (CONSTANT (MAXPAGESIZE) - 1)); . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE));

  .preinit_array     :
  {
    PROVIDE_HIDDEN (__preinit_array_start = .);
    KEEP (*(.preinit_array))
    PROVIDE_HIDDEN (__preinit_array_end = .);
  }

   . = ALIGN(. != 0 ? 64 / 8 : 1);

  .debug_pubtypes 0 : { *(.debug_pubtypes) }
Quuxplusone commented 8 years ago 
With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD
"hello world."

The current userland world build fails due to unhandled R_X86_64_TLSGD:

--- libc.so.7 ---
unrecognized reloc 19
cc: error: linker command failed with exit code 1 (use -v to see invocation)

The kernel build fails in the linker script parser:

--- kernel.full ---
linking kernel.full
) expected, but got ,
Quuxplusone commented 8 years ago 
(In reply to comment #10)
> With the change in http://reviews.llvm.org/D13637 LLD can now link a FreeBSD
> "hello world."
>
> The current userland world build fails due to unhandled R_X86_64_TLSGD:
>
> --- libc.so.7 ---
>
> unrecognized reloc 19
>
> cc: error: linker command failed with exit code 1 (use -v to see invocation)
>

Michael Spencer is working on it -- there's an initial patch under review so
that should be unlocked soon.

>
>
> The kernel build fails in the linker script parser:
>
> --- kernel.full ---
>
> linking kernel.full
>
> ) expected, but got ,

 http://reviews.llvm.org/D13668
Quuxplusone commented 8 years ago 
Update,

With the WIP -r patch (http://reviews.llvm.org/D14382) linking static libpam
produces a new error:

$ ld -o openpam_static_modules.o -r --whole-archive openpam_static.o
../modules/pam_chroot/libpam_chroot.a ../modules/pam_deny/libpam_deny.a [...]
FDE doesn't reference another section

If I'm able to investigate and produce a small reproduction case I will submit
a new PR.

Also, the lack of GNU symbol version support is now becoming a blocking issue
for making further progress on FreeBSD - our libc has some cases where it is
not usable without a symver-supporting linker.
Quuxplusone commented 8 years ago

Do you keep track of the progress? I'm interested in the numbers of packages we can and can't link with lld (with test success).

Quuxplusone commented 8 years ago 
> Do you keep track of the progress? I'm interested in the numbers of packages
we
> can and can't link with lld (with test success).

We're not yet at the point of trying to link the ports tree with lld -- I'm
working on the base system now.

After recent commits and the WIP -r patch I've been able to reduce the set of
hacks in my build tree to a small handful:

- default library search paths - ld.bfd includes /lib as a built-in search
path, for lld I need to specify it explicitly (in the libc.so DSO linker script)
- skip libpam/static_modules due to "FDE doesn't reference another section" as
mentioned in an earlier comment
- skip the i386 boot loaders due to lack of -N support in lld (proposed
implementation for the old elf linker in http://reviews.llvm.org/D968)
- skip the gdb build - it failed because lld does not provide an _etext symbol
- change -Tscript to -T script
- remove the EFI boot loaders because they require linker script functionality
not yet implemented
- skip uathload as it relies on an unusual use of -r (converting a binary blob
into an ELF object)

For now I'm not trying to build the kernel with lld.
Quuxplusone commented 8 years ago 
(In reply to comment #14)
> > Do you keep track of the progress? I'm interested in the numbers of
packages we
> > can and can't link with lld (with test success).
>
> We're not yet at the point of trying to link the ports tree with lld -- I'm
> working on the base system now.
>
> After recent commits and the WIP -r patch I've been able to reduce the set
> of hacks in my build tree to a small handful:
>
> - default library search paths - ld.bfd includes /lib as a built-in search
> path, for lld I need to specify it explicitly (in the libc.so DSO linker
> script)
> - skip libpam/static_modules due to "FDE doesn't reference another section"
> as mentioned in an earlier comment
> - skip the i386 boot loaders due to lack of -N support in lld (proposed
> implementation for the old elf linker in http://reviews.llvm.org/D968)
> - skip the gdb build - it failed because lld does not provide an _etext
> symbol
> - change -Tscript to -T script
> - remove the EFI boot loaders because they require linker script
> functionality not yet implemented
> - skip uathload as it relies on an unusual use of -r (converting a binary
> blob into an ELF object)

If you can open individual bugs for each one that would be awesome :-)
Quuxplusone commented 8 years ago 
> If you can open individual bugs for each one that would be awesome :-)

Unlike the recent PRs I've opened up I'm not convinced all of these are bugs
(in lld):

> - default library search paths - ld.bfd includes /lib as a built-in search
> path, for lld I need to specify it explicitly (in the libc.so DSO linker
> script)

The clang driver passes -L/usr/lib to ld and we expect to resolve -llib args
from there; this issue only happens because /usr/lib/libc.so is a linker script
that contains:

GROUP ( libc.so.7 libc_nonshared.a libssp_nonshared.a )

I think it should probably be:

GROUP ( /lib/libc.so.7 /usr/lib/libc_nonshared.a /usr/lib/libssp_nonshared.a )

> - skip libpam/static_modules due to "FDE doesn't reference another section"
> as mentioned in an earlier comment

This one indeed looks like an lld issue and I'll submit a PR once I have at
least some detail.

> - skip the i386 boot loaders due to lack of -N support in lld (proposed
> implementation for the old elf linker in http://reviews.llvm.org/D968)

-N should be easy to implement, but I'm not sure how useful or widely used it
is, and it seems reasonable for us to rework our boot components to avoid it.

> - skip the gdb build - it failed because lld does not provide an _etext
> symbol

PR 26729

> - change -Tscript to -T script

PR 26730
We can easily make this change in FreeBSD, but as we start trying to compile
large sets of packages with lld I'm sure this will come up again.

> - remove the EFI boot loaders because they require linker script
> functionality not yet implemented

Will submit a PR referencing specific missing functionality

> - skip uathload as it relies on an unusual use of -r (converting a binary
> blob into an ELF object)

While I believe -r support is important in lld I'm not sure I can argue for
this use, and will probably change the FreeBSD build to use objcopy instead.
Quuxplusone commented 8 years ago 
The set of workarounds I'm using to try building FreeBSD world with lld is in
my github FreeBSD tree: https://github.com/emaste/freebsd/tree/lld-buildworld
Other than those changes I'm testing with upstream FreeBSD and upstream lld.

The major outstanding issue is lack of expression support in linker scripts -
in my tree the kernel is being linked with ld.bfd for this reason. There are
also a few other components that are disabled or linked with ld.bfd.

I've built a VM image from this tree and large portions work; the system boots
to a login prompt, shells work, clang works, etc. Anything linked against libxo
(which includes ls and ps) segfaults - PR 26818. ps is used in many places in
the startup scripts and thus results in a lot of console spam.

Once PR 26818 is fixed I believe enough will be working that we can test lld in
a FreeBSD ports experimental run, trying to build the ~25K third party software
packages in the ports tree with lld.
Quuxplusone commented 8 years ago

Thank you very much for the updates. Linker script is indeed the next big one (and hopefully the last one), but before start working on it, I'm trying to support versioned symbols.

Quuxplusone commented 8 years ago

I'm trying to support versioned symbols.

Great.

For my immediate testing I'm building a full system w/o symver and the lack of support isn't holding me up -- I might have a couple of workarounds that I'll need to apply, but it's not a significant problem for buildworld (userland). Using a ld.bfd-linked kernel fine for testing.

Indeed, versioned symbol support will be very important for us if we're to provide a /usr/bin/ld.lld, in order to interoperate with system components -- particularly libc.so -- linked by GNU ld.

Quuxplusone commented 8 years ago

little note about:

  • skip libpam/static_modules due to "FDE doesn't reference another section" >as mentioned in an earlier comment

That should be fixed in r262590. That message was shown becauses eh_frame sections were proccessed during generating relocatable output and from that revision it is not happens anymore as well as this adds support for generating relocations against local symbols which were sections in that case.

Quuxplusone commented 8 years ago

In r262910, I add /usr and /usr/lib as default search paths. I think that should resolve one issue for FreeBSD.

Quuxplusone commented 8 years ago 
Changes have been committed to FreeBSD to address some of these issues:

* Library search paths, added to lld and then reverted. Now solved by leaving
the full paths in the /usr/lib/libc.so linker script:
https://svnweb.freebsd.org/changeset/base/296921

* ld -r to convert a binary object to ELF. Solved by rewriting the utility to
avoid having a compiled-in binary object:
https://svnweb.freebsd.org/changeset/base/296889

I'm now starting to test FreeBSD/arm64 (AArch64) buildworld with lld. It
currently fails when building libc:

/usr/local/aarch64-freebsd/bin/ld: getutxent.So(.debug_info+0x3c):
R_AARCH64_ABS64 used with TLS symbol udb
/usr/local/aarch64-freebsd/bin/ld: getutxent.So(.debug_info+0x59):
R_AARCH64_ABS64 used with TLS symbol uf
/usr/local/aarch64-freebsd/bin/ld: utxdb.So(.debug_info+0x5c): R_AARCH64_ABS64
used with TLS symbol futx_to_utx.ut
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.So(.debug_info+0x3d):
R_AARCH64_ABS64 used with TLS symbol __je_tsd_tls
/usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.So(.debug_info+0x12dc):
R_AARCH64_ABS64 used with TLS symbol __je_tsd_initialized
/usr/local/aarch64-freebsd/bin/ld: xlocale.So(.debug_info+0x404):
R_AARCH64_ABS64 used with TLS symbol __thread_locale
/usr/local/aarch64-freebsd/bin/ld: setrunelocale.So(.debug_info+0x3d):
R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale

Note that patches are open for arm64 TLS relocation work:

http://reviews.llvm.org/D16201
http://reviews.llvm.org/D17980
http://reviews.llvm.org/D18026
http://reviews.llvm.org/D18031
Quuxplusone commented 8 years ago 
(In reply to comment #16)
> > - skip the i386 boot loaders due to lack of -N support in lld (proposed
> > implementation for the old elf linker in http://reviews.llvm.org/D968)
>
> -N should be easy to implement, but I'm not sure how useful or widely used
> it is, and it seems reasonable for us to rework our boot components to avoid
> it.

So, just in case. We are not going to support -N, right ? (otherwise I would
take that).
Quuxplusone commented 8 years ago

Because -N option is for compatibility with old Unix systems that have died out in 80s or 90s. If you still want that behavior for some special case (such as bootloader or something), you should use a linker script instead.

Quuxplusone commented 8 years ago 
(In reply to comment #24)
> Because -N option is for compatibility with old Unix systems that have died
> out in 80s or 90s. If you still want that behavior for some special case
> (such as bootloader or something), you should use a linker script instead.

Yes, I assumed that I will be changing the FreeBSD build to implement what's
needed via a linker script.

I believe lld's linker script support is not yet fully able to implement -N's
functionality and I will continue to link these components with GNU ld until it
can.
Quuxplusone commented 8 years ago 
(In reply to comment #24)
> Because -N option is for compatibility with old Unix systems that have died
> out in 80s or 90s. If you still want that behavior for some special case
> (such as bootloader or something), you should use a linker script instead.

That's why I asked :)
Quuxplusone commented 8 years ago

Attempting to link FreeBSD/mips64 with lld reports:

warning: unknown argument: -EB

I do not yet know why there's an explicit big/little endian argument passed in by Clang.

Quuxplusone commented 8 years ago

It looks like lib/Driver/Tools.cpp always produces -m elf{32,64}{bt,lt}smip flags, so maybe -EB/EL are just redundant?

Quuxplusone commented 8 years ago

It looks like lib/Driver/Tools.cpp always produces -m elf{32,64}{bt,lt}smip flags, so maybe -EB/EL are just redundant?

Yes, I'm having trouble imagining how -EB/-EL could be used, where endianness is not already determined by either -m or the object file(s). For testing I've just added -EB and -EL as ignored options.

Quuxplusone commented 8 years ago

Moving to "ELF" bugzilla component (instead of generic "All Bugs" for LLD).

Quuxplusone commented 8 years ago 
Update on linking the FreeBSD kernel with -- following functionality used by
the FreeBSD/amd64 kernel link script not yet in lld:

- DATA_SEGMENT_RELRO_END
  https://reviews.llvm.org/D22676

- output section address assignment
  https://reviews.llvm.org/D22689

- AT (load address)
  https://reviews.llvm.org/D19272

- SIZEOF_HEADERS
  http://llvm.org/pr28688

- SORT

- EXCLUDE_FILE

- CONSTRUCTORS

obj file specification - e.g. KEEP (*crtbegin.o(.dtors))
Quuxplusone commented 8 years ago

One more issue for the FreeBSD/amd64 kernel linker script: expressions with symbols on the right hand side of an assignment

https://reviews.llvm.org/D22759

Quuxplusone commented 8 years ago 
status update at lld r280348 on x86-64, based on the workarounds in my test
tree:

issues / missing functionality in lld:
- [chars] wildcards in version scripts
- ~ (bitwise not) in linker script expressions
- -nostdlib, -f, -b binary options
- MIPS mxgot or multigot
- can't create dynamic relocation R_386_GOTOFF against readonly segment
  errors linking the 32-bit libraries

FreeBSD issues / issues needing triage:
- our system demangler produces non-standard output for some symbols, so
  extern "C++" in version scripts sometimes fails to match
- build for boot loader components needs to be updated to use linker scripts
  instead of obsolete options
- rescue (a multi-tool binary, like busybox) fails to build, investigation
  needed
- kernel links wtih lld but I have not tested it yet
Quuxplusone commented 8 years ago 
(In reply to comment #33)
> status update at lld r280348 on x86-64, based on the workarounds in my test
> tree:
>
> issues / missing functionality in lld:

+ Orphans handling I think. D23352
Quuxplusone commented 7 years ago 
There are currently two issues remaining when linking all of FreeBSD head world
+ kernel with lld head (r288670), on amd64:

1) PR31277 - lld segfault when compiling the 32-bit compat version of ldd
2) EFI loaders, when linked with LLD, either hang or report "Invalid Parameter."

The EFI loaders are built with a slightly convoluted set of steps:

a) linked as ELF files with debug

b) elfcopy/objcopy used to create a standalone debug file and stripped file
% objcopy --strip-debug loader.sym.full loader.sym

c) elfcopy/objcopy used to convert the stripped file to a PE/COFF EFI file
% SOURCE_DATE_EPOCH=1451606400  objcopy \
  -j .peheader -j .text -j .sdata -j .data  \
  -j .dynamic -j .dynsym -j .rel.dyn  \
  -j .rela.dyn -j .reloc -j .eh_frame \
  --output-target=efi-app-x86_64 loader.sym.full loader.efi

George R. reported errors from GNU objcopy at step (b), and success with a
manual build when skipping (b) and running (c) on the non-separated-debug
output from (a). I will submit a PR for the EFI loader issue once I have some
more detail and am convinced it is something that ought to have an LLD change.
Quuxplusone commented 7 years ago

Attached loader.zip (1298516 bytes, application/x-zip-compressed): loader repro

Quuxplusone commented 7 years ago 
(In reply to comment #35)
> George R. reported errors from GNU objcopy at step (b), and success with a
> manual build when skipping (b) and running (c) on the non-separated-debug
> output from (a).

Yes, I just retested loaders with different objcopy/elfcopy and -N flag.
Script for loader has next calls:

objcopy --strip-debug loader.sym.full loader.sym
objcopy \
  -j .peheader -j .text -j .sdata -j .data  \
  -j .dynamic -j .dynsym -j .rel.dyn  \
  -j .rela.dyn -j .reloc -j .eh_frame \
  --output-target=efi-app-x86_64 loader.sym loader.efi

My results using lld head (r288757):
1) Using GNU objcopy 2.17.50 [FreeBSD] 2007-07-03,
objcopy errors out:
 BFD: loader.sym: section `.dynstr' can't be allocated in segment 5
 objcopy: loader.sym: Bad value
 BFD: loader.sym: section `.dynstr' can't be allocated in segment 5

2) The same objcopy as above, but add -N (omagic). Loader works, no errors from
objcopy.

3) Using GNU objcopy (GNU Binutils) 2.27 it works both with or without -N !

4) Using elfcopy (elftoolchain HEAD FreeBSD svn:3500) instead of objcopy:
Either with or without -N loader hangs on startup, without any errors from
elfcopy side.

So my conclusion is that looks elfcopy is just broken, since objcopy 2.27 works
fine.
Just in case - my reproduce files with script for testing are in the post above.
Quuxplusone commented 7 years ago 
(In reply to comment #37)
>
> So my conclusion is that looks elfcopy is just broken, since objcopy 2.27
> works fine.
> Just in case - my reproduce files with script for testing are in the post
> above.

Thanks for this, it turns out it's just a plain bug in ELF Tool Chain's
elfcopy, now reported in https://sourceforge.net/p/elftoolchain/tickets/541/.
As far as I can tell no LLD changes are necessary to make this work. (I'd still
like to come back to the multiple .data sections and related differences vs.
GNU ld later on).
Quuxplusone commented 7 years ago 
(In reply to comment #38)
>
> Thanks for this, it turns out it's just a plain bug in ELF Tool Chain's
> elfcopy, now reported in
> https://sourceforge.net/p/elftoolchain/tickets/541/.

That is now fixed and imported as FreeBSD r310634, and I can boot using a LLD-
linked and elfcopy-converted loader.efi.

An issue remains with an LLD-linked legacy 32-bit BIOS boot path. I'm currently
bisecting building certain boot components with GNU ld vs LLD to determine
what's failing.
Quuxplusone commented 7 years ago 
(In reply to comment #39)
>
> An issue remains with an LLD-linked legacy 32-bit BIOS boot path. I'm
> currently bisecting building certain boot components with GNU ld vs LLD to
> determine what's failing.

This was due to an old assumption in FreeBSD's btxldr (boot code protected mode
wrapper) that ELF objects have two PT_LOAD segments. LLD uses an rodata segment
by default and so produced three PT_LOAD segments, and btxldr stopped
processing after the second. The fix is in https://reviews.freebsd.org/D8929

With that change applied I can link the entirety of the FreeBSD/amd64 base
system (userland world and kernel) with LLD.
Quuxplusone commented 7 years ago

W00t! What a milestone! Thanks Ed and everyone at the lld community!

How much of ports do we need to test before shipping an lld-freebsd alpha version for testing?

It would be good to have a Kvm image somewhere to give it a go.

Quuxplusone commented 7 years ago 
(In reply to comment #41)
> W00t! What a milestone! Thanks Ed and everyone at the lld community!

Thanks! I'm amazed at how far lld New-ELF has come in about a year and a half.

> How much of ports do we need to test before shipping an lld-freebsd alpha
> version for testing?

Rafael's done an initial experimental Poudriere FreeBSD package build with lld
head, and found almost 20K out of 26K ports built successfully. I'm now looking
at getting CI running to test this on an ongoing basis. But, I think we're at
the point where an experimental build makes sense.

> It would be good to have a Kvm image somewhere to give it a go.

We can quite easily produce VM images from FreeBSD-HEAD, which has
Clang/LLVM/LLDB/LLD 3.9.1 (less useful). I'll see about hacking something
together - either copying in an out-of-tree lld as with the Poudriere
experiment, or perhaps dim@ is planning to import a new LLVM snapshot in the
FreeBSD tree and I'll build from that.
Quuxplusone commented 7 years ago 
> With that change applied I can link the entirety of the FreeBSD/amd64 base
> system (userland world and kernel) with LLD.

That is truly awesome! I am almost looking forward to the end of my vacations
to get back to the poudriere run :-)

What is the process for getting patches into ports? By now I assume most issues
will be things like library order dependency which is better fixed in the
package.
Quuxplusone commented 7 years ago 
(In reply to comment #43)
>
> What is the process for getting patches into ports? By now I assume most
> issues will be things like library order dependency which is better fixed in
> the package.

I suggest that during development we collect patches in a local git repo -- for
example, I've started here for my Poudriere run
https://github.com/emaste/freebsd-ports/commits/ports-lld

The best case scenario is that we get any required changes committed upstream
first, and I've done that with the pkg change here:
https://github.com/freebsd/pkg/pull/1523 and bug report
https://github.com/freebsd/pkg/issues/1522

(ld.bfd can link pkg despite this issue so I've submitted bug 31495, but it's
really a bug in pkg.)

Assuming that the changes are not accepted upstream (or move too slowly, as
expected with libtool) then we can just submit a FreeBSD ports PR with the
desired patch.
Quuxplusone commented 7 years ago 
(In reply to comment #44)
> (In reply to comment #43)
> >
> > What is the process for getting patches into ports? By now I assume most
> > issues will be things like library order dependency which is better fixed in
> > the package.
>
> I suggest that during development we collect patches in a local git repo --
> for example, I've started here for my Poudriere run
> https://github.com/emaste/freebsd-ports/commits/ports-lld

Interesting. I will send you pull requests and open upstream bugs. How do you
get poudriere to use it? I thought it only had support for svn.
Quuxplusone commented 7 years ago 
(In reply to comment #45)
> Interesting. I will send you pull requests and open upstream bugs. How do
> you get poudriere to use it? I thought it only had support for svn.

Poudriere has support for fetching the ports tree from git (but not the base
system src tree, for some reason), but I'm not using that.

You can also configure Poudriere to build an from externally maintained ports
tree, using a command like:

% poudriere ports -c -F -f none -M /path/to/ports/tree -p lld

This info is from
https://github.com/freebsd/poudriere/wiki/use_system_ports_tree. It's not easy
to find and not clear from the normal Poudriere documentation (e.g. the man
pages).

Using this approach you can then update the ports tree any way you see fit and
then rerun.

In my Poudriere run perl5 failed to build due to dtrace relocs
  /usr/bin/ld: error: dtrace_mini.o:(.SUNW_dof+0x180): unrecognized reloc 8
and that took out 19743 downstream ports. (I knew many ports would have a
dependency chain passing through perl, but I'm surprised it's that extensive.)
Quuxplusone commented 7 years ago

In my Poudriere run perl5 failed to build due to dtrace relocs

This is the issue we've been discussing with markj; I believe dtrace support for perl was just recently enabled in the ports tree which would explain why Rafael didn't see it. I'll let my current run finish, and then restart when Mark's fix is available.

Quuxplusone commented 7 years ago 
(In reply to comment #47)
> > In my Poudriere run perl5 failed to build due to dtrace relocs
>
> This is the issue we've been discussing with markj; I believe dtrace support
> for perl was just recently enabled in the ports tree which would explain why
> Rafael didn't see it. I'll let my current run finish, and then restart when
> Mark's fix is available.

I actually hacked lld to ignore the broken dtrace created relocs. I posted the
patch to the list.
Quuxplusone commented 7 years ago 
(In reply to comment #48)
>
> I actually hacked lld to ignore the broken dtrace created relocs. I posted
> the patch to the list.

Yep, I saw that. I'm trying to look at this from the other side, so I want to
avoid applying hacks to lld and will apply WIP patches / hacks / etc. to the
FreeBSD ports tree for faults that appear to be FreeBSD (or other 3rd party)
issues.

It now looks like libtool is responsible for the majority of my failed /
skipped ports. Unless we really think we'll add "not GNU" and other hacks to
lld we're going to have to address libtool limitations upstream and in the
FreeBSD tree. I did look into libtool a few weeks ago, but unfortunately
haven't yet managed to produce a patch suitable for sending upstream.
Quuxplusone commented 7 years ago 
I've just added 31716 and 31762 to Depends On.
They are powerpc64 example issues. (I tend to
explore and report things for powerpc64 and
powerpc FreeBSD.)

I had not known about 23214's META for LLD and
had already listed those submittals in 25780's
META for clang targeting powerpc64 and powerpc
some time ago. (clang with the old system
binutils in FreeBSD is insufficient to allow
clang to be used as the system compiler
targeting powerpc64 without an external
binutils of some kind.)

At least for 25780 (clang's FreeBSD META) Ed
Maste has requested that I make additions to
that Depends On list myself. I expect that
would apply here as well.