Clang/LLVM TOT/12.x unable to build Python TOT

[Quuxplusone]

Bugzilla Link	PR49461
Status	NEW
Importance	P normal
Reported by	Aditya Atluri (aditya.gpu01@gmail.com)
Reported on	2021-03-05 13:29:39 -0800
Last modified on	2021-04-03 14:11:57 -0700
Version	trunk
Hardware	Macintosh MacOS X
CC	florian_hahn@apple.com, htmldeveloper@gmail.com, llvm-bugs@lists.llvm.org
Fixed by commit(s)
Attachments	build_llvm.sh (3575 bytes, text/x-sh) build_llvm.sh (3575 bytes, text/x-sh)
Blocks
Blocked by
See also

Created attachment 24595
Script to build clang and llvm

Hi,

I am trying to build tot python from https://github.com/python/cpython using
clang tot from: https://github.com/llvm/llvm-project

I am seeing the following error:

clang -dynamiclib -Wl,-single_module   -flto -Wl,-export_dynamic -g -fprofile-
instr-generate -undefined dynamic_lookup -Wl,-
install_name,/Users/aditya/.local/Python3.10.0a6/lib/libpython3.10.dylib -Wl,-
compatibility_version,3.10 -Wl,-current_version,3.10 -o libpython3.10.dylib
Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/parser.o
Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o
Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o
Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o
Objects/capsule.o Objects/cellobject.o Objects/classobject.o
Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o
Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o
Objects/genobject.o Objects/fileobject.o Objects/floatobject.o
Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o
Objects/iterobject.o Objects/listobject.o Objects/longobject.o
Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o
Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o
Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o
Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o
Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o
Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o
Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o
Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o
Python/ceval.o Python/codecs.o Python/compile.o Python/context.o
Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o
Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o
Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o
Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o
Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o
Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o
Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o
Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o
Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o
Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o
Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o
Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o
Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o
Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o
Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o
Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o
Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o
Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o
Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o
Modules/bufferedio.o Modules/textio.o Modules/stringio.o
Modules/faulthandler.o  Modules/_tracemalloc.o  Modules/symtablemodule.o
Modules/xxsubtype.o Python/frozen.o  -ldl   -framework CoreFoundation -v

clang version 13.0.0 (https://github.com/adityaatluri/llvm-project.git
ecdae5df7da03c56d72796c0b1629edd0995548e)

Target: x86_64-apple-darwin20.3.0

Thread model: posix

InstalledDir: /Users/aditya/.local/llvm13_x86_64/bin

 "/usr/bin/ld" /Users/aditya/.local/llvm13_x86_64/lib/clang/13.0.0/lib/darwin/libclang_rt.profile_osx.a -demangle -lto_library /Users/aditya/.local/llvm13_x86_64/lib/libLTO.dylib -dynamic -dylib -arch x86_64 -platform_version macos 11.0.0 11.1 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk -undefined dynamic_lookup -undefined dynamic_lookup -o libpython3.10.dylib -single_module -export_dynamic -install_name /Users/aditya/.local/Python3.10.0a6/lib/libpython3.10.dylib -compatibility_version 3.10 -current_version 3.10 Modules/getbuildinfo.o Parser/token.o Parser/pegen.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/dynload_shlib.o Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o Modules/errnomodule.o Modules/pwdmodule.o Modules/_sre.o Modules/_codecsmodule.o Modules/_weakref.o Modules/_functoolsmodule.o Modules/_operator.o Modules/_collectionsmodule.o Modules/_abc.o Modules/itertoolsmodule.o Modules/atexitmodule.o Modules/signalmodule.o Modules/_stat.o Modules/timemodule.o Modules/_threadmodule.o Modules/_localemodule.o Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o Modules/faulthandler.o Modules/_tracemalloc.o Modules/symtablemodule.o Modules/xxsubtype.o Python/frozen.o -ldl -framework CoreFoundation -sectalign __DATA __llvm_prf_cnts 0x4000 -sectalign __DATA __llvm_prf_data 0x4000 -lSystem /Users/aditya/.local/llvm13_x86_64/lib/clang/13.0.0/lib/darwin/libclang_rt.osx.a

Assertion failed: (It->second == &Insn && "Cached first special instruction is
wrong!"), function validate, file /Users/aditya/Code/github/src/llvm-
project/llvm/lib/Analysis/InstructionPrecedenceTracking.cpp, line 87.

clang-13: error: unable to execute command: Abort trap: 6

clang-13: error: linker command failed due to signal (use -v to see invocation)

How I built clang:
Attached script (get_llvm_os_all.sh) that i used to build clang for different
versions
I am using XCode 12.4 to build clang from source

How I built python:
Attached script (get_python.sh). I build openssl from source, if you already
have it, remove configure flag for openssl

Platform:
MacOS big sur 11.2.2
Xcode: Version 12.4 (12D4e)
Xcode clang:
Apple clang version 12.0.0 (clang-1200.0.32.29)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
TOT clang:
clang version 13.0.0 (https://github.com/llvm/llvm-project.git
ecdae5df7da03c56d72796c0b1629edd0995548e)
Target: x86_64-apple-darwin20.3.0
Thread model: posix
InstalledDir: /Users/aditya/.local/llvm13_x86_64/bin

Let me know if you need more information

[Quuxplusone]

Attached build_llvm.sh (3575 bytes, text/x-sh): Script to build clang and llvm

[Quuxplusone]

Attached build_llvm.sh (3575 bytes, text/x-sh): Script to build python

[Quuxplusone]

Could you try to apply https://reviews.llvm.org/D98183, re-build libLTO (target is LTO) and run ld as above, but with -mllvm -save-temps as extra options?

This should contain a bunch of ld-temp.* files, which are the intermediary files for LTO. Could you check if opt -O3 crashes for the generated ld-temp.0.3.import.bc ? Or llc for ld-temp.0.5.precodegen.bc.

If any of those crashes, it would be much easier to reproduce if you could share the bitcode files directly.

[Quuxplusone]

(In reply to Florian Hahn from comment #2)

Could you try to apply https://reviews.llvm.org/D98183, re-build libLTO (target is LTO) and run ld as above, but with -mllvm -save-temps as extra options?

This should contain a bunch of ld-temp.* files, which are the intermediary files for LTO. Could you check if opt -O3 crashes for the generated ld-temp.0.3.import.bc ? Or llc for ld-temp.0.5.precodegen.bc.

If any of those crashes, it would be much easier to reproduce if you could share the bitcode files directly.

You don't really need to apply the patch, you can also just pass -save-temps to ld without the patch and it should generate a lto.bc. It would be great if you could check that against opt -O3 and the `lto.opt.bc` against llc

[Quuxplusone]

Hi, I did not understand "It would be great if you could check that against opt -O3 and the *lto.opt.bc against llc". Able to apply patch and generate ld-temp.0.3.import.bc file