Open InBetweenNames opened 6 years ago
@InBetweenNames I would use it on my router too, I think at the time the gcc LTO toolchain wasn't very mature and few were able too make much use of it, particularly embedded* Linux where there would be most interest. Without that buy-in the kernel devs weren't going to let the patches in.
Perhaps resurrecting the patch set and getting it working again could be successful now that lto support is pretty ubiquitous in distros and most embedded devs must be using it by now for their user space.
Seems some remnants of those patches are still in the kernel (notably DISABLE_LTO so it doesn't use it for vdso), so I tried with 4.19.1. Formerly used scripts/gcc-ld but didn't work for me so I used gold. I doubt it's accomplishing anything built this way (size barely changed with other defaults). Despite using gcc-ar, was also complaining about the lto plugin unless -ffat-lto. Patchset used to use -fwhole-program too but that didn't work. Nonetheless, thought I'd do the crazy thing and build the kernel with:
make -j8 AR=gcc-ar NM=gcc-nm LD=ld.gold KCFLAGS="-march=native -O3 -falign-functions=32 -fipa-pta -fno-semantic-interposition -fgraphite-identity -floop-nest-optimize -flto=8 -ffat-lto-objects" DISABLE_LTO=-fno-lto
Which.. worked.. and booted fine. I am now the proud owner of a kernel that 30% bigger than before, probably not faster, and set out to kill my dog, but thankfully running in QEMU away from my dog. Edit: well, removing LTO with the same options does make it like 10% even bigger.
It might be interesting to compare the speed of some syscall- / kernel-bound workloads when successfully built with LTO. Anyone with an idea on how to start benchmarking our gains or losses?
Not sure, but if you check the kernel mailing list plenty of those benchmarks have been done in the past. I remember seeing pretty big gains with LTO, but not sure if those reflected into any gain for daily usage. Some more info about how to benchmark the kernel: https://github.com/graysky2/kernel_gcc_patch
One thing about LTO is you have to build as many of your models into the kernel as possible... so it knows what it can eliminate when linking... so you get the biggest gains on a completely static kernel (this of course breaks somethings that load firmware etc... some of that you can work around by building in the blobs though).
Andi Kleen rebased his LTO patches for the Kernel on 4.20 recently. I've tried it out but had no luck and several module errors along the way. Nevertheless, you can find these patches here: https://github.com/andikleen/linux-misc/tree/lto-420-1
^ Didn't experiment much but gave it a quick try and it built fine for me with my configuration and CONFIG_LTO=y
which auto-adds -flto -fno-fat-lto-objects
. Didn't try a generic one and I use almost no modules which, as stated in the other above post, is better suited for a LTO kernel anyway.
Looks like it's using the gcc-ld script and working properly. I do have gold as my default linker (been using it even for kernel).
I imagine it may make more of a difference on a less-lean kernel, but my resulting 4.20 kernel is about 1% smaller than my old, didn't try to boot and also no idea for any performance gains.
@ionenwks I'm trying to replicate the steps on a gentoo system to build an LTO'd kernel. However, I always error out on the linking portion: /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/../../../../x86_64-pc-linux-gnu/bin/ld: error: arch/x86/kernel/head_64.o: requires unsupported dynamic reloc 11; recompile with -fPIC
I have added this flag to the base KBUILD_CFLAGS but to no effect. I also have ld.gold enabled by default.
What version of GCC and binutils are you using? Did you make any configurations to the Makefile from Andi Kleen's repo?
Hmm... I tried again both with the lto-420-1 branch from back then along with same configuration and the newer lto-5.1-3, and I'm getting the same errors as you now (using gcc 8.3.0 and ld.bfd 2.32).
Not sure what I was using back then but looking at the date I assume I was on gcc 8.2 and binutils 2.30 I think? It's only something I tried real quick, I had no intention to stick with that for now (or boot it).
Edit: Retried with gold as default (switched back to bfd a while ago), doesn't work either, not with current toolchain anyway. Edit2: And no, I hadn't made any changes, used as-is.
@ionenwks thank you for taking the time to check through the issue! I was afraid it was a toolchain version issue, so I wonder if this is a reportable bug? I'm going to take some time today and check if its a gcc or binutils issue. Edit: I'm throwing some more configuration testing into this mess. Found this article over on the patch list: https://patchwork.kernel.org/patch/10000627/
I was able to build 5.0-1 successfully, however I did not test it and the system it was on it now gone.
-fPIC
would cause reloc .text errors if it was built with visibility=hidden or ssp(but the Makefile already filters that). Maybe -flinker-output=rel would make sense here, but I couldn't get the syntax correct. ~because parts of the kernel build are still static, and static objects aren't able to find PIC references~. If anyone knows his full patchset without a kernel tree that'd be really helpful.
@jiblime You can find his patchset on the kernel mailing list but it won't really help: https://lkml.org/lkml/2017/11/27/1052
THIN_ARCHIVES
was a config option that was removed in 4.19+. It went around the supposed issue of ld -r
. But, I've narrowed it down to a ld
issue of some sort. There are kernel patches that let you fPIC the code but they aren't working for me yet.
@Promaethius Thanks for the link. I'm currently trying to edit arch/x86/entry/vdso/Makefile to work. At the very bottom you can try appending flags after ${LD} but nothing has worked for me, even the options to specifically suppress the error.
I went and checked a regular kernel and I noticed that it's normal(?) for a hidden symbol to be there.
Both comands ran were readelf vclock_gettime.o -s
5.1-3 LTO:
Symbol table '.symtab' contains 25 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS vclock_gettime.c
2: 0000000000000000 0 SECTION LOCAL DEFAULT 1
3: 0000000000000000 0 SECTION LOCAL DEFAULT 3
4: 0000000000000000 0 SECTION LOCAL DEFAULT 4
5: 0000000000000000 174 FUNC LOCAL DEFAULT 1 do_hres
6: 0000000000000000 0 SECTION LOCAL DEFAULT 5
7: 0000000000000000 0 SECTION LOCAL DEFAULT 7
8: 0000000000000000 0 SECTION LOCAL DEFAULT 8
9: 0000000000000000 0 SECTION LOCAL DEFAULT 10
10: 0000000000000000 0 SECTION LOCAL DEFAULT 11
11: 0000000000000000 0 SECTION LOCAL DEFAULT 12
12: 0000000000000000 0 SECTION LOCAL DEFAULT 14
13: 0000000000000000 0 SECTION LOCAL DEFAULT 15
14: 0000000000000000 0 SECTION LOCAL DEFAULT 17
15: 0000000000000000 0 SECTION LOCAL DEFAULT 19
16: 0000000000000000 0 SECTION LOCAL DEFAULT 20
17: 0000000000000000 0 SECTION LOCAL DEFAULT 18
18: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND vvar_vsyscall_gtod_data
19: 00000000000000b0 111 FUNC GLOBAL DEFAULT 1 __vdso_clock_gettime
20: 00000000000000b0 111 FUNC WEAK DEFAULT 1 clock_gettime
21: 0000000000000120 98 FUNC GLOBAL DEFAULT 1 __vdso_gettimeofday
22: 0000000000000120 98 FUNC WEAK DEFAULT 1 gettimeofday
23: 0000000000000190 16 FUNC GLOBAL DEFAULT 1 __vdso_time
24: 0000000000000190 16 FUNC WEAK DEFAULT 1 time
readelf: Warning: compressed section '.debug_str' is corrupted
5.2.8 kernel:
Symbol table '.symtab' contains 27 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND
1: 0000000000000000 0 FILE LOCAL DEFAULT ABS vclock_gettime.c
2: 0000000000000000 0 SECTION LOCAL DEFAULT 1
3: 0000000000000000 0 SECTION LOCAL DEFAULT 3
4: 0000000000000000 0 SECTION LOCAL DEFAULT 4
5: 0000000000000000 392 FUNC LOCAL DEFAULT 1 do_hres
6: 0000000000000000 0 SECTION LOCAL DEFAULT 5
7: 0000000000000000 0 SECTION LOCAL DEFAULT 7
8: 0000000000000000 0 SECTION LOCAL DEFAULT 8
9: 0000000000000000 0 SECTION LOCAL DEFAULT 10
10: 0000000000000000 0 SECTION LOCAL DEFAULT 11
11: 0000000000000000 0 SECTION LOCAL DEFAULT 12
12: 0000000000000000 0 SECTION LOCAL DEFAULT 14
13: 0000000000000000 0 SECTION LOCAL DEFAULT 15
14: 0000000000000000 0 SECTION LOCAL DEFAULT 17
15: 0000000000000000 0 SECTION LOCAL DEFAULT 19
16: 0000000000000000 0 SECTION LOCAL DEFAULT 20
17: 0000000000000000 0 SECTION LOCAL DEFAULT 18
18: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND vvar_vsyscall_gtod_data
19: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND hvclock_page
20: 0000000000000000 0 NOTYPE GLOBAL HIDDEN UND pvclock_page
21: 0000000000000190 102 FUNC GLOBAL DEFAULT 1 __vdso_clock_gettime
22: 0000000000000190 102 FUNC WEAK DEFAULT 1 clock_gettime
23: 0000000000000200 98 FUNC GLOBAL DEFAULT 1 __vdso_gettimeofday
24: 0000000000000200 98 FUNC WEAK DEFAULT 1 gettimeofday
25: 0000000000000270 16 FUNC GLOBAL DEFAULT 1 __vdso_time
26: 0000000000000270 16 FUNC WEAK DEFAULT 1 time
readelf: Warning: compressed section '.debug_str' is corrupted
looks to be of interest. Does this mean there needs to be more debug information built in?
@jiblime I found this on the gcc site today: https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-fuse-linker-plugin-916
When a file is compiled with -flto without -fuse-linker-plugin, the generated object file is larger than a regular object file because it contains GIMPLE bytecodes and the usual final code (see -ffat-lto-objects. This means that object files with LTO information can be linked as normal object files; if -fno-lto is passed to the linker, no interprocedural optimizations are applied. Note that when -fno-fat-lto-objects is enabled the compile stage is faster but you cannot perform a regular, non-LTO link on them.
I've witnessed Andi Kleen's patchset passing -fno-fat-lto-objects
without -fuse-linker-plugin
. Will test this theory later today. This could explain why readelf is returning corruption, but pardon my ignorance if that's not the case.
@Promaethius
I've witnessed Andi Kleen's patchset passing -fno-fat-lto-objects without -fuse-linker-plugin
That explains why he uses -fwhole-program and and -fipa-cp-clone, since collect2 would be used instead of a linker. I'm assuming he's doing that for compatibility, as GCC documentation claims it's likely to increase code size vs. bfd/gold. I wonder if GentooLTO would be able to do something better...
I believe it's a glibc issue. I've upgraded to sys-libs/glibc-2.30::gentoo and have been able to get past it. Currently recompiling since paravirtualization options, not sure which, causes it to error.
https://sourceware.org/ml/libc-alpha/2019-08/msg00029.html
* The dynamic linker no longer refuses to load objects which reference
versioned symbols whose implementation has moved to a different soname
since the object has been linked. The old error message, symbol
FUNCTION-NAME, version SYMBOL-VERSION not defined in file DSO-NAME with
link time reference, is gone.
It emits a warning, I'm still not sure why since Andi Kleen filters LTO out of it from what I can tell.
CC arch/x86/entry/vdso/vdso32-setup.o - due to target missing LDS arch/x86/entry/vdso/vdso.lds - due to target missing AS arch/x86/entry/vdso/vdso-note.o - due to target missing CC arch/x86/entry/vdso/vclock_gettime.o - due to target missing In file included from ./arch/x86/include/asm/vgtod.h:5, from arch/x86/entry/vdso/vclock_gettime.c:15: arch/x86/entry/vdso/vclock_gettime.c: In function ‘do_hres’: ./include/linux/compiler.h:182:26: warning: array subscript 1 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds] 182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \ | ^~~~~~~~~~~~~~~~~~~~ ./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’ 193 | __READ_ONCE_SIZE; | ^~~~~~~~~~~~~~~~ arch/x86/entry/vdso/vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’ 37 | extern u8 hvclock_page | ^~~~~~~~~~~~ In file included from ./arch/x86/include/asm/vgtod.h:5, from arch/x86/entry/vdso/vclock_gettime.c:15: ./include/linux/compiler.h:182:26: warning: array subscript 2 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds] 182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \ | ^~~~~~~~~~~~~~~~~~~~ ./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’ 193 | __READ_ONCE_SIZE; | ^~~~~~~~~~~~~~~~ arch/x86/entry/vdso/vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’ 37 | extern u8 hvclock_page | ^~~~~~~~~~~~ CC arch/x86/entry/vdso/vgetcpu.o - due to target missing VDSO arch/x86/entry/vdso/vdso64.so.dbg - due to target missing OBJCOPY arch/x86/entry/vdso/vdso64.so - due to target missing HOSTCC arch/x86/entry/vdso/vdso2c - due to target missing VDSO2C arch/x86/entry/vdso/vdso-image-64.c - due to target missing CC arch/x86/entry/vdso/vdso-image-64.o - due to target missing LDS arch/x86/entry/vdso/vdso32/vdso32.lds - due to target missing CC arch/x86/entry/vdso/vdso32/vclock_gettime.o - due to target missing In file included from ./arch/x86/include/asm/vgtod.h:5, from arch/x86/entry/vdso/vdso32/../vclock_gettime.c:15, from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31: arch/x86/entry/vdso/vdso32/../vclock_gettime.c: In function ‘do_hres’: ./include/linux/compiler.h:182:26: warning: array subscript 1 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds] 182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \ | ^~~~~~~~~~~~~~~~~~~~ ./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’ 193 | __READ_ONCE_SIZE; | ^~~~~~~~~~~~~~~~ In file included from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31: arch/x86/entry/vdso/vdso32/../vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’ 37 | extern u8 hvclock_page | ^~~~~~~~~~~~ In file included from ./arch/x86/include/asm/vgtod.h:5, from arch/x86/entry/vdso/vdso32/../vclock_gettime.c:15, from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31: ./include/linux/compiler.h:182:26: warning: array subscript 2 is outside array bounds of ‘u8[1]’ {aka ‘unsigned char[1]’} [-Warray-bounds] 182 | case 8: *(__u64 *)res = *(volatile __u64 *)p; break; \ | ^~~~~~~~~~~~~~~~~~~~ ./include/linux/compiler.h:193:2: note: in expansion of macro ‘__READ_ONCE_SIZE’ 193 | __READ_ONCE_SIZE; | ^~~~~~~~~~~~~~~~ In file included from arch/x86/entry/vdso/vdso32/vclock_gettime.c:31: arch/x86/entry/vdso/vdso32/../vclock_gettime.c:37:11: note: while referencing ‘hvclock_page’ 37 | extern u8 hvclock_page | ^~~~~~~~~~~~ AS arch/x86/entry/vdso/vdso32/note.o - due to target missing AS arch/x86/entry/vdso/vdso32/system_call.o - due to target missing AS arch/x86/entry/vdso/vdso32/sigreturn.o - due to target missing VDSO arch/x86/entry/vdso/vdso32.so.dbg - due to target missing OBJCOPY arch/x86/entry/vdso/vdso32.so - due to target missing VDSO2C arch/x86/entry/vdso/vdso-image-32.c - due to target missing CC arch/x86/entry/vdso/vdso-image-32.o - due to target missing
So as I understand, it would be a huge issue to have a textrel in a/the vdso because it'd be a vulnerability in a security feature. Gentoo's wiki actually has a guide on finding and fixing textrels: https://wiki.gentoo.org/wiki/Hardened/Textrels_Guide
But hopefully there's no need to recreate anything. While the vdso*.so files have a textrel flag marked on them, scanelf -T
shows that there isn't anything that would point to it.
Glibc 2.29, GCC 9.1.0
TYPE PAX PERM ENDIAN STK/REL/PTL TEXTREL RPATH BIND TEXTRELS FILE
scanelf: scanelf_file_textrels(): ELF is missing relocation information
scanelf: scanelf_file_textrels(): ELF vdso32.so has TEXTREL markings but doesnt appear to have any real TEXTREL's !?
ET_DYN PeMRxS 0755 LE --- --- R-X TEXTREL - LAZY vdso32.so
It did also emit this, though:
arch/x86/kernel/dumpstack.o: warning: objtool: show_regs.cold()+0x16: sibling call from callable instruction with modified stack frame
arch/x86/kernel/dumpstack.o: warning: objtool: show_regs()+0x0: stack state mismatch: cfa1=7+24 cfa2=7+8
So it looks like it can be possible, but definitely experimental and not a daily driver for myself. I'm going to be grabbing GCC 9.2 now so I won't be getting to it anytime soon (btw, I added 20G of swap with -j5 and it still failed, dammit), but if Glibc 2.30 is the fix, I think it'd be worth a shot to try using this kernel for testing.
If you were to use a linker instead of collect2 you can run replace -fwhole-program
with fuse-linker-plugin
in scripts/Makefile.lto as Gnu documentation states it's best not to use the former with the latter. Optimizations that would also help LTO specifically would be -fdevirtualize-at-ltrans
and the -fgraphite-identity -floop-nest-optimize
options. I've used these along other flags to compile and run my kernel, but if the linking stage is too much the process will overflow and it'll end.
What's interesting is that his newest version (as far as I can tell) lacks explicit linker usage but his older versions use -fuse-linker-plugin
. So I could be wrong in assuming that removing -fwhole-program
is the right way to go.
Andi Kleen's lto-5.7-2 branch branch builds and I am currently running it. I've applied the 5.7.14 patch, Gentoo distro patches, and a few other misc. patches with no rejects.
Notes:
nouveau does not build due to the command to building it being too long for shell
I wasn't able to load any modules, not when booted in or through my initrd. Used make mod2yesconfig
to convert modules to being built in. [Correction 1 below]
./scripts/Makefile.lto contains the settings for how LTO is done upon the kernel. I appended -flto-compression-level=9
to LTO_CFLAGS
In the same file, TMPDIR sets the building directory in the kernel directory instead of /tmp to prevent OOM. I copied the kernel to /var/tmp/portage instead because there are massive writes on disk during linking.
The primary demographic of LTO'd kernels seems to be embedded systems
The size of my LTO'd kernel is 22M, modules folder is 800K. Vs. my normal kernel at 11M and modules folder at 71M
Semi-related:
GCC 10's -O2 might be slightly slower than GCC 9's -O2
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96337#c15
Inliner changes was not targetting to make compile time faster and
compiled code slower. It was intended to reflect more closely modern C++
codebases and get faster binaries (at -O2 and -O2 -flto) without
regressing in code sizes. In fact more inlining happens and thus we
needed to optimize inliner code carefully to avoid regressions with LTO.
If you have a -march=znver1/znver2 processor and run x86_64 multilib, rebuilding the current GCC 10.2.0 would mean a nice performance boost with this patch:
Refer to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95435
Correction 1: I incorrectly assumed modules weren't supported with -flto
. While building everything into the kernel alleviated the issues, namely framebuffer and Logitech USB support, kernel compilation time was too long and I prefer being able to reload modules. The likely culprit in module failure was TRIM_UNUSED_KSYMS
and possibly dracut defaulting to --strip
the generated initrd; can't say for certain yet. I didn't get around to testing it enough but now I am able to load amdgpu
in my initrd as usual instead of compiling it in.
* It feels fast, that counts
Can you describe in what way?
Cheers for the gcc links too
oooh, imma test
@jiblime Could you list the patches applied? All are from gentoo's ebuild?
@jiblime Could you list the patches applied? All are from gentoo's ebuild?
I haven't built it yet, but this patch applies fine to gentoo-sources-5.8 (just a diff from the lto-5.8.0-1 branch)
https://gist.github.com/telans/728b63dd07c41c9ca6e2ca3d4431db8e
Doesn't build for me unfortunately, lots of:
/usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: ./.tmp_vmlinux.kallsyms1.mJXteD.ltrans123.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
also, there's no 5.7.14 patch
https://cdn.kernel.org/pub/linux/kernel/v5.x/patch-5.7.14.xz
that patch is already applied... nvm, messed up something, ended with upstream master somehow... this will impact my sdd most def
@telans No problem. The first thing I noticed was my dmesg timestamps were lower than usual :p ideally I'll set up a phoronix benchmark to have actual data.
relocation R_X86_64_32S
Are you using ld.gold as your default linker? The Linux kernel needs either GCC/ld.bfd or Clang/ld.lld. https://github.com/InBetweenNames/gentooLTO/issues/338
sys-devel/gcc-10.2.0::gentoo was built with the following:
USE="(cxx) fortran graphite lto (multilib) nls nptl objc openmp pch pgo sanitize ssp zstd (-ada) -d -debug -doc (-fixed-point) -go (-hardened) -jit (-libssp) -objc++ -objc-gc -pie -systemtap -test -vanilla -vtv" ABI_X86="(64)"
sys-devel/binutils-2.34-r2::gentoo was built with the following:
USE="gold multitarget nls plugins static-libs -default-gold -doc -test" ABI_X86="(64)"
@barolo https://github.com/jiblime/linux-misc/commits/lto-5.7-prjc-r3 You can pull the patches from here or clone the single branch and build off that. The CPPC patch doesn't work for me, so I leave it off just in case it would case me to fail to boot. It's a bit messy, I'm still not the greatest at making clean commits. I chose the 5.7-2 branch instead of 5.8 because I wanted to try the Project C scheduler (previously named BMQ, now abbreviated prjc). I'll try the 5.8 branch sometime.
I generally download a vanilla tarball from kernel.org (v5.7, v5.8, etc) and apply the Gentoo patches and incremental patches afterwards. That way I don't have to worry about rejected patches as often
@jiblime thanks for the branch, made it much easier for me. Compiling
compiled almost cleanly for me, didn't take that long too, had a bunch of "-Wstringop-overflow" warnings for Bluetooth module. Didn't boot for me with error related to scsi. With modules builtin it is 20M , modules dir i 1M I have nvme and amdgpu on that box, gonna try to strip it a bit more
Narrowed it down, hidpp/logitech's stuff makes it crash, and it doesn't switch to amdgpu output @jiblime it seems like you\ve had similar issues, how did you solve them? Edit. Cleaned it a bit, built amdgpu, bluetooth, and logitech hidpp as modules, the remaining issue seems to be that framebuffer isn't being switched during boot
Are you using ld.gold as your default linker? The Linux kernel needs either GCC/ld.bfd or Clang/ld.lld.
Nope, using ld.bfd ( or at least I haven't changed it.)
sys-devel/gcc-10.2.0::gentoo was built with the following:
USE="(cxx) fortran graphite lto (multilib) nls nptl openmp pch pgo (pie) sanitize ssp vtv zstd (-ada) -d -debug -doc (-fixed-point) -go (-hardened) (-jit) (-libssp) -objc -objc++ -objc-gc -systemtap -test -vanilla" ABI_X86="(64)"
sys-devel/binutils-2.34-r2::gentoo was built with the following:
USE="gold nls plugins -default-gold -doc -multitarget -static-libs -test" ABI_X86="(64)"
Forcing LD=ld.bfd
doesn't change anything either. I thought it might have been an issue with ripping a patch from the lto-5.8-1
branch, however, the branch too builds with the same relocation errors
Same issue with lto-5.7-2
Update, managed to run it and reach the desktop. The issue was with building all modules in.
So I took my working config as base, used genkernel and made sure that it runs without LTO enabled first, then enabled LTO and booted into desktop successfully.
Ended with a bunch of drivers disabled, most importantly for network and sata, luckily my main is a pcie one.
Each failed module had disagrees about version of symbol module_layout
in dmesg, gonna investigate it now.
Edit. It seems that all of those are modules that weren't built in, so it seems that initramfs isn't working for me
Edit2. I'm typing from it, had to recompile it cleanly, cleaned it a bit and built some stuff in, module loading doesn't seem to work as I still got two of those disagrees...
warnings
Can't really compare it yet, since it seems to use diff schedulers than I had with zen kernel, and spends more time at lower frequencies, would have to bench it properly to test it seriously.
I can already tell though that building that kernel is significantly faster under it
My gut tells me it has something to do with the -fPIE flag
On Sun, Aug 9, 2020, 3:27 AM Greg Shuiske notifications@github.com wrote:
Update, managed to run it and reach the desktop. The issue was with building all modules in. So I took my working config as base, used genkernel and made sure that it runs without LTO enabled first, then enabled LTO. Ended with a bunch of drivers disabled, most importantly for network and sata, luckily my main is a pcie one. Each module had disagrees about version of symbol module_layout in dmesg, gonna investigate it now.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/InBetweenNames/gentooLTO/issues/90#issuecomment-671029236, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFFNN3MKRMCYOAXWIEI7BLR7ZTWVANCNFSM4EN5L3PQ .
@Promaethius I've solved that by having those with warnings changed to built-in, It's running fine so far, gonna bench it with something now. My whole kernel with inbuilt stuff is 10 MB, with useless 4MB initramfs, for gaming desktop
when linux 5.7.14 w/ the lto-5.7-2
using ld.bfd
from binutils 2.35
and gcc-10.1.0
i'm getting countless errors all telling me to recompile with -fPIE
similar to what @telans has mentioned:
DESCEND objtool
CALL scripts/atomic/check-atomics.sh
CALL scripts/checksyscalls.sh
CHK include/generated/compile.h
GEN .version
CHK include/generated/compile.h
UPD include/generated/compile.h
CC init/version.o
AR init/built-in.a
LDFINAL vmlinux.o
kernel/bpf/core.c: In function 'bpf_patch_insn_single':
kernel/bpf/core.c:442:3: warning: writing 8 bytes into a region of size 0 [-Wstringop-overflow=]
442 | memcpy(prog->insnsi + off, patch, sizeof(*patch));
| ^
./include/linux/filter.h:550:20: note: at offset 0 to object 'insnsi' with size 0 declared here
550 | struct bpf_insn insnsi[0];
| ^
MODPOST vmlinux.o
MODINFO modules.builtin.modinfo
GEN modules.builtin
LDFINAL .tmp_vmlinux.kallsyms1
/usr/bin/ld: arch/x86/kernel/head_64.o: relocation R_X86_64_32S against symbol `early_top_pgt' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/entry_64.o: relocation R_X86_64_32S against symbol `cpu_tss_rw' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/vdso/vma.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/vdso/vdso32-setup.o: relocation R_X86_64_32S against symbol `vdso_image_32' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/entry/entry_64_compat.o: relocation R_X86_64_32S against symbol `cpu_tss_rw' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/kvm/vmx/vmenter.o: relocation R_X86_64_32S against symbol `kvm_rebooting' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/realmode/init.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/kernel/acpi/wakeup_64.o: relocation R_X86_64_32S against symbol `saved_magic' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/kernel/ftrace_64.o: relocation R_X86_64_32S against symbol `ftrace_trace_function' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/crypto/aesni-intel_asm.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/crypto/aesni-intel_avx-x86_64.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/crypto/ghash-clmulni-intel_asm.o: relocation R_X86_64_32S against `.rodata.cst16.bswap_mask' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/power/hibernate_asm_64.o: relocation R_X86_64_32S against symbol `saved_context' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans0.ltrans.o: relocation R_X86_64_32S against hidden symbol `cpu_hw_events' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans1.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans2.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans3.ltrans.o: relocation R_X86_64_32S against symbol `em_bsf' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans4.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans5.ltrans.o: relocation R_X86_64_32S against hidden symbol `cpu_bit_bitmap' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans6.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans7.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans8.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans9.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans10.ltrans.o: relocation R_X86_64_32S against `.data..read_mostly' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans11.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans12.ltrans.o: relocation R_X86_64_32S against hidden symbol `sig_sicodes.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans13.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans14.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans15.ltrans.o: relocation R_X86_64_32S against hidden symbol `rcu_data.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans16.ltrans.o: relocation R_X86_64_32S against hidden symbol `tk_fast_mono.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans17.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans18.ltrans.o: relocation R_X86_64_32S against hidden symbol `trace_types_lock' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans19.ltrans.o: relocation R_X86_64_32S against hidden symbol `ftrace_common_fields.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans20.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans21.ltrans.o: relocation R_X86_64_32S against hidden symbol `init_task' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans22.ltrans.o: relocation R_X86_64_32S against hidden symbol `contig_page_data' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans23.ltrans.o: relocation R_X86_64_32S against hidden symbol `vma_interval_tree_augment_rotate.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans24.ltrans.o: relocation R_X86_64_32S against hidden symbol `vmap_area_list' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans25.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans26.ltrans.o: relocation R_X86_64_32S against hidden symbol `__per_cpu_offset' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans27.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans28.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans29.ltrans.o: relocation R_X86_64_32S against `.data..percpu' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans30.ltrans.o: relocation R_X86_64_32S against hidden symbol `bit_wait_table.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans31.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans32.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans33.ltrans.o: relocation R_X86_64_32S against hidden symbol `tty_drivers' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans34.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans35.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans36.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans37.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans38.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans39.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans40.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans41.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans42.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans43.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans44.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans45.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans46.ltrans.o: relocation R_X86_64_32S against `.rodata..c_jump_table' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans47.ltrans.o: relocation R_X86_64_32S against hidden symbol `queue_io_timeout_entry.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans48.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans49.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans50.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans51.ltrans.o: relocation R_X86_64_32S against hidden symbol `dev_attr_boot_vga.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans52.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans53.ltrans.o: relocation R_X86_64_32S against hidden symbol `v86d_path.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans54.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans55.ltrans.o: relocation R_X86_64_32S against hidden symbol `_ctype' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans56.ltrans.o: relocation R_X86_64_32S against hidden symbol `_ctype' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans57.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans58.ltrans.o: relocation R_X86_64_32S against hidden symbol `vc_cons' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans59.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans60.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans61.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans62.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans63.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans64.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans65.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans66.ltrans.o: relocation R_X86_64_32S against hidden symbol `execlists_submit_request.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans67.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans68.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans69.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans70.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans71.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans72.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans73.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans74.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans75.ltrans.o: relocation R_X86_64_32S against hidden symbol `device_ktype.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans76.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans77.ltrans.o: relocation R_X86_64_32S against hidden symbol `dma_buf_fops.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans78.ltrans.o: relocation R_X86_64_32S against hidden symbol `scsi_host_type.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans79.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans80.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans81.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans82.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans83.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans84.ltrans.o: relocation R_X86_64_32S against hidden symbol `ant_toggle_lookup.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans85.ltrans.o: relocation R_X86_64_32S against hidden symbol `dev_attr_manufacturer.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans86.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans87.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans88.ltrans.o: relocation R_X86_64_32S against hidden symbol `scsi_host_type.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans89.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans90.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans91.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans92.ltrans.o: relocation R_X86_64_32S against `.data' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans93.ltrans.o: relocation R_X86_64_32S against hidden symbol `tpacpi_all_drivers.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans94.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans95.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans96.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans97.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans98.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans99.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans100.ltrans.o: relocation R_X86_64_32S against hidden symbol `rtnl_af_ops.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans101.ltrans.o: relocation R_X86_64_32S against hidden symbol `offload_base.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans102.ltrans.o: relocation R_X86_64_32S against hidden symbol `loggers.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans103.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans104.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans105.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans106.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans107.ltrans.o: relocation R_X86_64_32S against `.text' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans108.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans109.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans110.ltrans.o: relocation R_X86_64_32S against `.bss' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans111.ltrans.o: relocation R_X86_64_32S against hidden symbol `ptype_lock.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans112.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans113.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans114.ltrans.o: relocation R_X86_64_32S against hidden symbol `init_uts_ns' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans115.ltrans.o: relocation R_X86_64_32S against hidden symbol `swevent_htable.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans116.ltrans.o: relocation R_X86_64_32S against hidden symbol `ieee802_1d_to_ac' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans117.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans118.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans119.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans120.ltrans.o: relocation R_X86_64_32S against hidden symbol `tcf_action_policy.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans121.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans122.ltrans.o: relocation R_X86_64_32S against hidden symbol `con2fb_map.lto_priv.0' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans123.ltrans.o: relocation R_X86_64_32S against `.rodata' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans124.ltrans.o: relocation R_X86_64_32S against symbol `_stext' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans125.ltrans.o: relocation R_X86_64_32S against hidden symbol `init_mm' can not be used when making a PIE object
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans126.ltrans.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: ./.tmp_vmlinux.kallsyms1.KgBLDA.ltrans127.ltrans.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/lib/getuser.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: arch/x86/lib/putuser.o: relocation R_X86_64_32S against symbol `current_task' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: drivers/base/firmware_loader/builtin/iwlwifi-4965-2.ucode.gen.o: warning: relocation in read-only section `.builtin_fw'
collect2: error: ld returned 1 exit status
make: *** [Makefile:1111: vmlinux] Error 1
My gut tells me it has something to do with the -fPIE flag
I agree. It looks like PIE may be the main reason why there are build issues. I'm unsure of what implementation is best to remove auto PIE just for the kernel.
When testing out with gcc -m32 {-fpie ; -fPIE ; -Wl,-pie} file.c -o m32
.
$ gcc -m32 -Wl,-pie file.c -o m32
$ file m32
ELF 32-bit LSB pie executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 3.2.0, not stripped
Only -Wl,-pie
worked, meaning -Wl,-no-pie
or -Wl,-no-PIE
would be ideal? Except it error'd on me, and file
tells me that -m64 compiles aren't PIE even with those set. Could be a problem with my toolchain and inexpertise.
I'd suggest just rebuilding GCC without pie, though you may need to rebuild multiple parts of your toolchain that have been emerged with pie, namely binutils/glibc/libtool, but some other suggestions you can try first:
make KCFLAGS="-fno-pie -fno-PIE -Wl,-no-pie" LDFLAGS_MODULE="Wl,-no-pie" -j"$(grep processor /proc/cpuinfo | wc -l)"
or
In ./scripts/Makefile.build, I believe these are inherited for all builds unless the Makefile has a specific override.
EXTRA_AFLAGS :=
EXTRA_CFLAGS :=
EXTRA_CPPFLAGS :=
EXTRA_LDFLAGS :=
You can also append V=1 to make and get verbose output to see what flags are being used if that may help:
Some more info on PIE: https://wiki.gentoo.org/wiki/Hardened/Toolchain#Position_Independent_Executables_.28PIEs.29 https://stackoverflow.com/questions/34519521/why-does-gcc-create-a-shared-object-instead-of-an-executable-binary-according-to https://stackoverflow.com/questions/43367427/32-bit-absolute-addresses-no-longer-allowed-in-x86-64-linux
Some issues I found with the v5.8-1 linux-misc LTO branch:
compiled almost cleanly for me, didn't take that long too, had a bunch of "-Wstringop-overflow" warnings for Bluetooth module. Didn't boot for me with error related to scsi. With modules builtin it is 20M , modules dir i 1M I have nvme and amdgpu on that box, gonna try to strip it a bit more
I got those too, bluetooth and a couple other things. Apparently it's related to GCC aggressively inlining for no reason, and it's been a thing for a while. I believe the the fix is to directly tell GCC not to inline those modules. I think it would be a good idea to make a list of modules to tell GCC not to inline.
I'm sure there's a kernel doc on that somewhere...Linus really hates -O3 for its inlining lol. Sidenote, GCC 10 no longer enables -funroll-loops at any opt level. Side-side note: RDRAND is re-enabled, rerun cpuid2cpuflags to see if your list has changed
I have GCC built without -fPIE, That would explain why I'm succeding. I still can't load modules though
@barolo Perhaps try KALLSYMS_ALL
?
@jiblime are you suggesting that rebuilding of a toolchain w/o pie
as a possible solution to my problem here? i don't have any modules.
@jblime I'm yet to try KALLSYMS_ALL
but I've already added -fdevirtualize-at-ltrans -fgraphite-identity -floop-nest-optimize
, checked that it's actually added during compilation and it works! Running it right now, now to complete the trifecta PGO would be needed xD
added
-fdevirtualize-at-ltrans -fgraphite-identity -floop-nest-optimize
Benchmarks would be awesome if you could
@barolo Mirroring a phoronix kernel benchmark would probably be the easiest with benches such as: https://www.phoronix.com/scan.php?page=article&item=linux-50-sliding&num=2
@telans I was just about to ask... xD Could someone recommend a simple bench to test the kernel? ( possibly not Phoronix.. )
Edit. Averaged Motionmark 1.1 is 20% faster than under Zen kernel, both set to performance mode. I will try the same kernel now just without LTO/graphite, to have a valid comparison.
@mssx86
Yes. Ideally your toolchain and @system
. If that is too time consuming it'd probably be enough to just rebuild GCC and binutils. There was a discussion about it: https://github.com/InBetweenNames/gentooLTO/issues/261 It has a paper on the overhead that PIE introduces and how to unmask the (pie) useflag.
@barolo I did too. I'm surprised it's running haha! Check out this patch too, I think it might make sense: https://raw.githubusercontent.com/jiblime/clear-ck-gentoo-sources/5.8-lto-exp/misc/0004-Add-option-to-use-fno-semantic-interposition-and-fde.patch
I also added -fno-inline-functions -fno-inline-functions-called-once --param=large-stack-frame-growth=100
to my KCFLAGS, no more overflow warnings.
And I agree @telans, a standard would be nice to measure. It would be a lot easier to quantify/benchmark changes in the kernel vs. something system-wide. These are from perf but I wouldn't put any weight in it
If only Phoronix suites were so quick 😛
@jiblime ~so you have fully working 5.8 now? any differences from 5.7 regarding LTO?~ Could you explain those additional flags?
It's definitely quicker, you can see, feel it even without benches. The difference is so significant in execution time that I thought it's the different scheduler lowering peak cpu usage initially Are you doing phoronix? That would save me from some pain...
Edit. I'm on zen patched 5.8 lto kernel now, for fair comparison since Zen is my kernel of choice due to fsync. Doing some phoronix tests now
@jiblime can confirm that after rebuilding gcc with --disable-default-pie
, i managed to build and boot with the lto'd kernel. thanks a bunch.
Same here, sys-devel/gcc without (pie) works a treat. I did rebuild @system
just to be safe too.
I also encountered the same issue with config SINGLE_LINK
@jiblime. However, my kernel isn't any larger than before (no external modules aside from nvidia) at 9.3M vs the old at 9.5M
Boots fine but I'm coming across nvidia: disagrees about version of symbol module_layout
. I did rebuild multiple time with the kernel loaded/unloaded etc. Even when lto is disabled from the kernel config options the error appears. The 'fix' for me was to remove the lto patch I mentioned above.
Looks like this might be as far as I can go with this until I get an AMD card or something along those lines. Any other suggestions? Otherwise I'm not sure what to do.
Thanks all
Could you explain those additional flags?
@barolo -fno-inline-functions
disables automatic inlining on any possible function which was the default -O2 for GCC9. The second one probably isn't helpful to disable, the third is a reduction from a max of +1000% to +100% for stack growth caused by inlining. I chose those because overly aggressive inlining in some of the kernel's code only makes it bigger with no benefit and if I understand, large stacks in the kernel will mean some instructions won't fit in the CPU cache -> slower
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=49194 https://lwn.net/Articles/166172/
@telans I've found that rebuilding the kernel will remove some symbols from Module.symvers. What I did to fix that was backup my .config, run make mrproper
and move the config back. After rebuilding the kernel a new Module.symvers will be generated. I also have MODVERSIONS
and MODULE_SRCVERSION_ALL
off.
I think external kernel modules are affected by what the kernel was built with. Maybe try adding -ffat-lto-objects
to x11-drivers/nvidia-drivers?
Finally managed to get things sorted out and have some sane results: it's a Zen 5.8 -O3, native
vs Zen 5.8 -O3, native, graphite, LTO
@telans
[ it's a 4 core , AMD APU ]
I don't know what's going on with a single threaded OpenGL but I could tell the difference even without that bench... The results for a non-Zen kernel are even worse
run
make mrproper
and move the config back. After rebuilding the kernel a new Module.symvers will be generated
Thanks! I believe that's what fixed it. Currently booted now with no issues.
Test parameters:
GCC 10.2.0
binutils 2.35
glibc 2.30-r9
Ran in tty
CONFIG_GENERIC_CPU=y
(-mtune=generic, -march not set)
LTO patch, genpatches 1000-4567 (none of which affect performance)
Notes about testing:
Phoronix Test Suite v9.8.0 (Nesodden)
pts/stress-ng requires sys-libs/libapparmor
to link -lapparmor for tests
7-8% deviation on CPU Cache for both results, removed in later testing
Kernel 5.8.0, both -O2 Baseline: no-lto Result: lto
Size comparisons, using lz4 compression no-LTO -O2: 15939744 LTO -O2: 15464160 ~3% size decrease
Summary: clear performance benefit to using an LTO kernel
Kernel 5.8.0, both LTO
Baseline: LTO -O2
Result: LTO -O3 -fno-inline-functions
Note: ~3% deviation on memory copying, unsure why
Summary: LTO -O3 with -fno-inline-functions
performs worse against LTO -O2, and performs better or the same as no-LTO -O2
Kernel 5.8.0, both LTO Baseline: LTO -O2 Result: LTO -O3
Size comparisons, using lz4 compression no-LTO -O2: 15939744 LTO -O2: 15464160 LTO -O3: 16665088 ~4.4% size increase against no-LTO -O2, ~7.4% increase compared against LTO -O2
Summary: fno-inline-functions
has an observable impact on performance. LTO -O2 performs marginally better than LTO -O3 overall, and is smaller too.
It'd be better if I had more tests to reduce variation, also tested the effect of setting -march. Would GCC have been able to take advantage of my processor cache better? I found out recently that if you don't set -march=native, these defaults will be set: --param=l1-cache-line-size=32
& --param=l1-cache-size=64
unless user specified.
I'll run those same benchmarks later, however, currently perf
is giving me approximately 20% lower results on an lto kernel. Using KCFLAGS=-fno-inline-functions -fno-inline-functions-called-once --param=large-stack-frame-growth=100 -fdevirtualize-at-ltrans -fgraphite-identity -floop-nest-optimize
but I'll try with default flags.
After some rebuilding, I've found -fno-inline-functions
is enough to stop the warnings when using -O3. Still need to test if -O3 is faster, with the inlining flags I mentioned LTO -O3 is actually slower than LTO -O2 😞
Successful building/booting summary:
-fPIE
MODVERSIONS
and MODULE_SRCVERSION_ALL
set to NO
in kernel config if you want module loadingKALLSYMS_ALL
set to YES
and also do make mrproper
to have a fresh Module.symvers
CRYPTO_DEV_CCP_DD
[ an amd thing ] WILL crash your kernel if built inSINGLE_LINK
and KALLSYMS_SINGLE
set to OFF
or build will fail[ that's what's working for me ]
@jiblime got everything?
I find it interesting that there hasn't been more push to build the kernel using LTO. I've found a couple of mailing list threads about it, including a patchset to let it happen, but there wasn't a lot of interest upstream. I've created this issue as a way to track what the current LTO progress in the kernel is, and possibly even add some patchsets to let it happen. I know I'd for sure use it on my router if I could with OpenWRT.