EugeneZelenko
commented
7 months ago Could you please try 18 or main branch?
Open Cerberios opened 7 months ago
EugeneZelenko
commented
7 months ago Could you please try 18 or main branch?
Cerberios
commented
7 months ago Yes, sure just need compile bunch of packages. Will report asap
alexey-bataev
commented
7 months ago Yes, sure just need compile bunch of packages. Will report asap
Hi, thanks for the report. Compiler saves 2 files - sh script and the preprocessed source code file, which cause the crass (the names are reported in the end of the crash report). Could you, please, copy them and attach them to this bug report? It will help to investigate the crash and will save lots of time.
dememax
commented
6 months ago @alexey-bataev What are those files? At least, what are their extensions exactly? I'm attaching more longer file with my crash. firefox-llvm-17-crash.txt
dememax
commented
6 months ago By searching in Gentoo bugs, I've found https://bugs.gentoo.org/925101 And the link to https://github.com/llvm/llvm-project/issues/84062 from there.
alexey-bataev
commented
6 months ago @alexey-bataev What are those files? At least, what are their extensions exactly? I'm attaching more longer file with my crash. firefox-llvm-17-crash.txt
Can you try to invoke the compiler manually and add -Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll? I assume it should look like this:
/usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols-Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.llIt should produce repro.ll file, that causes the crash
dememax
commented
6 months ago Yes, it will take a while.
In fact, I've solved my problem by switching off -pgo, but I can rebuild to have locally the broken build, it is reproducible.
Can you contact me in person to instruct in the case this command will be incomplete or fail?
alexey-bataev
commented
6 months ago Yes, it will take a while. In fact, I've solved my problem by switching off
-pgo, but I can rebuild to have locally the broken build, it is reproducible. Can you contact me in person to instruct in the case this command will be incomplete or fail?
Sure, let's see if this one works
dememax
commented
6 months ago Here:
maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll
x86_64-pc-linux-gnu-clang++-17: warning: argument unused during compilation: '-mllvm -print-module-scope' [-Wunused-command-line-argument]
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>> stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)
maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # ls -lt
total 20
-rw-r--r-- 1 portage portage 2392 Apr 19 19:37 stub.o
-rw-r--r-- 1 portage portage 66 Apr 19 19:35 libipcclientcerts.so.symbols
-rw-r--r-- 1 portage portage 6464 Apr 19 18:59 backend.mk
-rw-r--r-- 1 portage portage 631 Apr 19 18:59 MakefileWith -v:
maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll -v
clang version 17.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/17/bin
Configuration file: /etc/clang/x86_64-pc-linux-gnu-clang++.cfg
System configuration file directory: /etc/clang
Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/14
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
x86_64-pc-linux-gnu-clang++-17: warning: argument unused during compilation: '-mllvm -print-module-scope' [-Wunused-command-line-argument]
"/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack/ld.lld" --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -shared -o libipcclientcerts.so /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/14/crtbeginS.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/14 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/lib -L/lib -L/usr/lib -plugin-opt=mcpu=skylake -plugin-opt=O3 -plugin-opt=thinlto -plugin-opt=-function-sections=1 -plugin-opt=-data-sections=1 -z relro -z now -z defs --gc-sections -h libipcclientcerts.so stub.o --real-linker /usr/lib/llvm/17/bin/ld.lld -plugin-opt=-import-instr-limit=10 -plugin-opt=-import-hot-multiplier=30 -lpthread -O1 --as-needed --undefined-version --compress-debug-sections=zlib -rpath=/usr/lib64/firefox --enable-new-dtags -z noexecstack -z text -z relro -z now -z nocopyreloc -rpath-link /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -rpath-link /usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a --version-script libipcclientcerts.so.symbols -plugin-opt=-print-on-crash -plugin-opt=-print-on-crash-path=repro.ll -lstdc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-pc-linux-gnu/14/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../lib64/crtn.o
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>> stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)In fact, seeing multilib in this output, I've got two hosts with Gentoo, and on one host it compiles, but on an other - it fails.
The difference is in gentoo profiles:
So, maybe the key of the problem is this multilib/no- problem.
dememax
commented
6 months ago @alexey-bataev should we switch to #84062 ? This issue is closed.
alexey-bataev
commented
6 months ago I reopened it, we can continue discussion here. Will check your info on Monday.
12101111
commented
6 months ago So, maybe the key of the problem is this multilib/no- problem.
My system is musl based and don't support multilib.
Since the workaround disables gpu acceleration when pgo, this bug may be related to the crashed process writing incorrect pgo profile files
alexey-bataev
commented
6 months ago Sorry, provided a bit incorrect flags. Here is the right one:
/usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -Wl,-plugin-opt=-print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.llNeed to use -Wl,-plugin-opt= instead of -mllvm for -print-module-scope just like for all other options
dememax
commented
6 months ago maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -Wl,-plugin-opt=-print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>> stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -Wl,-plugin-opt=-print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll ld.lld: error: undefined symbol: __llvm_profile_set_timestamp >>> referenced by stub.cpp >>> stub.o:(C_GetFunctionList) x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)
Check if there is a file repro.ll
dememax
commented
6 months ago @alexey-bataev Sorry, I haven't added, but no repro.ll.
alexey-bataev
commented
6 months ago So, maybe it crashes on another linking command. Need to rebuild the firefox with the correct flags (-Wl,-plugin-opt=)
dememax
commented
6 months ago So, maybe it crashes on another linking command. Need to rebuild the firefox with the correct flags (-Wl,-plugin-opt=)
Unfortunately, I don't have enough knowledge to do this. I've never touched firefox build system.
alexey-bataev
commented
6 months ago So, maybe it crashes on another linking command. Need to rebuild the firefox with the correct flags (-Wl,-plugin-opt=)
Unfortunately, I don't have enough knowledge to do this. I've never touched firefox build system.
No, I don't ask you to do it. Can you reproduce a crash manually?
dememax
commented
6 months ago What exactly should I run? ld.lld looks fine for me: ld.lld--help.txt ldd-ld.lld.txt
alexey-bataev
commented
6 months ago I'm just asking if you can reproduce the crash manually. If you run /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols does it crash at all or not?
dememax
commented
6 months ago We do a lot of nose. Maybe it will better to switch in PM? Yes, I run it manually.
alexey-bataev
commented
6 months ago We do a lot of nose. Maybe it will better to switch in PM? Yes, I run it manually.
Ok, where do you want to discuss it?
dememax
commented
6 months ago Status update on our experimentations with Alex.
After a little investigation, we found the problematic command (each option is on its own line for visibility):
/usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17
-U_FORTIFY_SOURCE
-D_FORTIFY_SOURCE=2
-fstack-protector-strong
-fstack-clash-protection
-fno-sized-deallocation
-fno-aligned-new
-march=native
-fno-exceptions
-fPIC
-fno-rtti
-ffunction-sections
-fdata-sections
-fno-exceptions
-fno-math-errno
-pthread
-O3
-fomit-frame-pointer
-funwind-tables
-shared
-Wl,-z,defs
-Wl,--gc-sections
-Wl,-h,libxul.so
-o libxul.so
/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/toolkit/library/build/libxul_so.list
-B
/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack
-Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld
-flto=thin
-Wl,-plugin-opt=-import-instr-limit=10
-Wl,-plugin-opt=-import-hot-multiplier=30
-lpthread
-Wl,-O1
-Wl,--as-needed
-Wl,--undefined-version
-Wl,--compress-debug-sections=zlib
-Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags
-fuse-ld=lld
-Wl,-z,noexecstack
-Wl,-z,text
-Wl,-z,relro
-Wl,-z,now
-Wl,-z,nocopyreloc
-fstack-protector-strong
-fstack-clash-protection
-Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin
-Wl,-rpath-link,/usr/lib
../../../js/src/build/libjs_static.a
../../../build/pure_virtual/libpure_virtual.a
/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libgkrust.a
../../../security/sandbox/linux/libmozsandbox.so
../../../config/external/gkcodecs/libgkcodecs.so
../../../config/external/lgpllibs/liblgpllibs.so
../../../config/external/sqlite/libmozsqlite3.so
../../../widget/gtk/mozgtk/libmozgtk.so
../../../widget/gtk/mozwayland/libmozwayland.so
-Wl,--version-script,libxul.so.symbols
-licui18n
-licuuc
-licudata
-lresolv
-laom
-ldav1d
-lasound
-lrt
-lX11
-lXcomposite
-lXdamage
-lXext
-lXfixes
-lXrandr
-lXrender
-lpthread
-ldl
-lc
-lffi
-lplds4
-lplc4
-lnspr4
-lz
-lm
-lssl3
-lsmime3
-lnss3
-lnssutil3
-lfreetype
-lfontconfig
-lgdk-3
-lharfbuzz
-lpangocairo-1.0
-lpango-1.0
-lgtk-3
-latk-1.0
-lcairo
-lcairo-gobject
-lgdk_pixbuf-2.0
-lglib-2.0
-lgobject-2.0
-lgio-2.0
-lgraphite2
-lpng16
-lwebp
-lwebpdemux
-levent
-lvpx
-lpixman-1
-ldbus-1
-lxcb-shm
-lX11-xcb
-lxcb
-lXcursor
-lXiAlex proposed to add these options to the command:
-Wl,-plugin-opt=-print-on-crash
-Wl,-plugin-opt=-print-module-scope
-Wl,-plugin-opt=-print-on-crash-path=repro.llThe crash was reproduced, but unfortunately, the repro.ll file wasn't produced. The output was:
ld.lld: warning: Linking two modules of different target triples: '/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libgkrust.a(nsstring-e605d9a4b5ee90ce.nsstring.a86a7024c2d63334-cgu.0.rcgu.o at 155447734)' is 'x86_64-unknown-linux-gnu' whereas '/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/toolkit/library/build/../../../netwerk/protocol/http/Unified_cpp_protocol_http4.o' is 'x86_64-pc-linux-gnu'
LLVM ERROR: too many signal callbacks already registered
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
#0 0x00007f5db61b02d3 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdb02d3)
#1 0x00007f5db61adf14 llvm::sys::RunSignalHandlers() (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdadf14)
#2 0x00007f5db61ae096 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdae096)
#3 0x00007f5db4e6fca0 (/lib64/libc.so.6+0x39ca0)
#4 0x00007f5db4ec3e3c (/lib64/libc.so.6+0x8de3c)
#5 0x00007f5db4e6fbf6 gsignal (/lib64/libc.so.6+0x39bf6)
#6 0x00007f5db4e588b8 abort (/lib64/libc.so.6+0x228b8)
#7 0x00007f5db5c02570 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x802570)
#8 0x00007f5db60e61cc llvm::report_fatal_error(char const*, bool) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xce61cc)
#9 0x00007f5db61ae753 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdae753)
#10 0x00007f5db9d0d0be llvm::PrintCrashIRInstrumentation::registerCallbacks(llvm::PassInstrumentationCallbacks&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x490d0be)
#11 0x00007f5db9d12821 llvm::StandardInstrumentations::registerCallbacks(llvm::PassInstrumentationCallbacks&, llvm::AnalysisManager<llvm::Module>*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x4912821)
#12 0x00007f5db7c2029f (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x282029f)
#13 0x00007f5db7c226a2 llvm::lto::opt(llvm::lto::Config const&, llvm::TargetMachine*, unsigned int, llvm::Module&, bool, llvm::ModuleSummaryIndex*, llvm::ModuleSummaryIndex const*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x28226a2)
#14 0x00007f5db7c23f8e (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2823f8e)
#15 0x00007f5db7c24dcd llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2824dcd)
#16 0x00007f5db7c0a906 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x280a906)
#17 0x00007f5db7c17713 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2817713)
#18 0x00007f5db60a73e4 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xca73e4)
#19 0x00007f5db6144c66 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd44c66)
#20 0x00007f5db61452dd (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd452dd)
#21 0x00007f5db4ec2080 (/lib64/libc.so.6+0x8c080)
#22 0x00007f5db4f372ac (/lib64/libc.so.6+0x1012ac)
Failed to hack libxul.so: basic_ios::clear: iostream error
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)
maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/toolkit/library/build # echo $?
1Next, Alex proposed to use other options instead of -plugin-opt=-print-*:
-Wl,-plugin-opt=save-temps
-Wl,-plugin-opt=-opt-bisect-limit=-1The crash was reproduced, but it wasn't enough to analyze the source of the problem.
Then, we tried to see if we could reproduce the crash with different values of -plugin-opt=-opt-bisect-limit option.
There was a hypothesis that the command starts to fail at some point, and we could find this value.
But the command fails with different values of the limit, and not in a monotonous way.
torvic9
commented
6 months ago I can confirm the same (or very similar) issue when compiling Firefox 125.x with PGO and LTO.
It happens both with clang17 and clang18 running an up-to-date Arch Linux installation.
The error seems to happen when compiling (not linking) the Firefox source file dom/canvas/DrawTargetWebGl.cpp [1] .
I tried passing -fno-tree-vectorize -fno-slp-vectorize to the whole dom/canvas source files, which made the build proceed a little further, but ultimately still crashing at one of the last steps.
I haven't tried yet passing these parameters to the whole build and currently I do not have access to my build machine, so for the moment I can't provide the exact info.
My CFLAGS are the default Arch CFLAGS [2] except for -march=x86-64-v3 instead of the default -march/-mcpu flags.
[1] https://hg.mozilla.org/releases/mozilla-release/file/tip/dom/canvas/DrawTargetWebgl.cpp [2] https://gitlab.archlinux.org/archlinux/packaging/packages/pacman/-/blob/main/makepkg.conf
alexey-bataev
commented
6 months ago I can confirm the same (or very similar) issue when compiling Firefox 125.x with PGO and LTO.
It happens both with clang17 and clang18 running an up-to-date Arch Linux installation.
The error seems to happen when compiling (not linking) the Firefox source file
dom/canvas/DrawTargetWebGl.cpp[1] .I tried passing
-fno-tree-vectorize -fno-slp-vectorizeto the wholedom/canvassource files, which made the build proceed a little further, but ultimately still crashing at one of the last steps.I haven't tried yet passing these parameters to the whole build and currently I do not have access to my build machine, so for the moment I can't provide the exact info.
My CFLAGS are the default Arch CFLAGS [2] except for
-march=x86-64-v3instead of the default -march/-mcpu flags.[1] https://hg.mozilla.org/releases/mozilla-release/file/tip/dom/canvas/DrawTargetWebgl.cpp [2] https://gitlab.archlinux.org/archlinux/packaging/packages/pacman/-/blob/main/makepkg.conf
If the compiler crashes at source file, it should save the shell script and the preprocessed source file. The source file itself does not help, need the preprocessed output, generated by the comnpiler
alexey-bataev
commented
6 months ago Finally found the reproducer and was able to reproduce the crash with 17.0.4 and 18.1.5. But the crash does not reproduce with the trunk, looks like the issue is fixed already and it will be fixed in llvm 19.0
torvic9
commented
6 months ago The issue does NOT appear if I append
-fno-tree-vectorize -fno-slp-vectorize
to the CFLAGS.
Build finishes normally without errors.
@alexey-bataev, thanks for the feedback, maybe we can find out what commit fixed this issue and have it backported to the still active 18.x branch?
EDIT: It also needs -march=x86-64, x86-64-v3 doesn't work either.
Also, I couldn't get Cross+FullLTO to work but that is a different issue.
alexey-bataev
commented
6 months ago The issue does NOT appear if I append
-fno-tree-vectorize -fno-slp-vectorizeto the CFLAGS.
Build finishes normally without errors.
@alexey-bataev, thanks for the feedback, maybe we can find out what commit fixed this issue and have it backported to the still active 18.x branch?
I tried to identify the fix, but there are too many potential candidates with too many dependencies. I can try to prepare a fix but it won't be a backport, it will be completely separate patch
Hello there! I am not a programmer, but got this error - please, investigate, may be it will help to make product better. I used sys-devel/llvm 17.0.6 + polly
System uname: Linux-6.8.4-gentoo-simple-gcc-x86_64-13th_Gen_Intel-R-_Core-TM-_i5-1335U-with-glibc2.38 python 3.11.9-final-0, default/linux/amd64/23.0/split-usr/hardened, gcc-13, glibc-2.38-r11, 6.8.4-gentoo-simple-gcc x86_64Please, let me know what kind of another logs (settings) I should provide.
Thanks!