Closed ghost closed 9 months ago
This is because there are two versions of libcxx in use. This exact problem also occurs when there are multiple versions of libcxx in use.
A simple way to see the problem is making a shell with llvmPackages_16.libcxx
in it. As it turns out, llvmPackages_16.libcxx
is different from the one in stdenv
, and you get two different libcxx-es in $NIX_LDFLAGS
. We also include jemalloc. Running clang++ hello.cxx -ljemalloc; ./a.out
then segfaults. In fact it segfaults on exactly the same line of code as @a-n-n-a-l-e-e: this one, where encoding
is a null function pointer and the code jumps to it.
I don't know how stdenv could include a libcxx other than the one in llvmPackages_XX, but I'm sure it's going to break a bunch more things than just jemalloc.
Using the current head of the release-23.11
branch (please note I am using aarch64, not x86_64 as OP was):
# shell.nix
let
nixpkgs = builtins.fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/e4e2a842524fae32efac9649aeb1edfed7e7ae72.tar.gz";
sha256 = "18aqndvq9x5vhyqzrs167s0jilii41fy5bdbyjx1ar2l3gy5p26x";
};
pkgs = import nixpkgs { config = {}; };
in
pkgs.mkShell {
packages = [
pkgs.jemalloc
pkgs.llvmPackages_16.libcxx
pkgs.llvmPackages_16.libcxxabi
];
}
$ nix-shell
[nix-shell:~/.../jemalloc-libc++-test]$ echo $NIX_LDFLAGS
# newlines added
-L/nix/store/1r9jhkzfz6q55qciyla3mz7nmh2w6cmz-jemalloc-5.3.0/lib
-L/nix/store/zq45ffwwcy2mw8mnnj14532gma3dsl05-libcxx-16.0.6/lib
-L/nix/store/1hr6xc1r3y6i9n6bkvci6a7kscv6cx8h-libcxxabi-16.0.6/lib
-L/nix/store/vf7k7l7ny35g1nmninaqjbryl2xqkag5-libcxx-16.0.6/lib
-L/nix/store/x622mzi1g7v7nh7zq5r8d0nxbhrg9px3-libcxxabi-16.0.6/lib
-L/nix/store/l92ls1vba62zsywdj81hlgmgx5adn6bn-compiler-rt-libc-16.0.6/lib
-L/nix/store/bj25mj5bpvbx03gqzf7hbpk6n7w4khr6-libobjc-11.0.0/lib
-L/nix/store/1r9jhkzfz6q55qciyla3mz7nmh2w6cmz-jemalloc-5.3.0/lib
-L/nix/store/zq45ffwwcy2mw8mnnj14532gma3dsl05-libcxx-16.0.6/lib
-L/nix/store/1hr6xc1r3y6i9n6bkvci6a7kscv6cx8h-libcxxabi-16.0.6/lib
-L/nix/store/vf7k7l7ny35g1nmninaqjbryl2xqkag5-libcxx-16.0.6/lib
-L/nix/store/x622mzi1g7v7nh7zq5r8d0nxbhrg9px3-libcxxabi-16.0.6/lib
-L/nix/store/l92ls1vba62zsywdj81hlgmgx5adn6bn-compiler-rt-libc-16.0.6/lib
-L/nix/store/bj25mj5bpvbx03gqzf7hbpk6n7w4khr6-libobjc-11.0.0/lib
Note:
-L/nix/store/zq45ffwwcy2mw8mnnj14532gma3dsl05-libcxx-16.0.6/lib
and -L/nix/store/vf7k7l7ny35g1nmninaqjbryl2xqkag5-libcxx-16.0.6/lib
(otool -l
on libjemalloc shows it is using the former)There are a few differences in nix derivation show
for each of the libcxxs. Maybe this will give some kind of clue. For the one jemalloc links (/nix/store/zq45ff...):
# newlines added
"nativeBuildInputs": "/nix/store/4xivbfq2sf0mcgql4j6ly93jwxjb2n31-cmake-3.27.7
/nix/store/k3s4yv7swx52vhpd7z5jhymn0ji5g2ni-ninja-1.11.1
/nix/store/7b0rz3bnx7msw2wawkv1hhn5lqf1b0wi-python3-3.11.6
/nix/store/0m03fp4cj5vhr34a8rvwr1y1y74fjbd7-fix-darwin-dylib-names-hook",
For llvmPackages_16.libcxx
(/nix/store/vf7k7l7...):
"nativeBuildInputs": "/nix/store/1l2hahnabzw0rmrmsxryxchf6lazghz9-cmake-minimal-3.27.7
/nix/store/azfh76jsad19zlxd1mh0b79lhl75x9rp-ninja-1.11.1
/nix/store/sxhmccj49jphf2n1nhybc9a4l9s4qkf0-python3-minimal-scproxy-3.11.6
/nix/store/rdd6cn1r40m1zd704d7lrm2abkkl5flm-fix-darwin-dylib-names-hook",
And finally, what happens when you try to link a hello world C program with jemalloc in that shell:
$ echo 'int main() {}' > hello.cxx
$ clang++ hello.cxx -ljemalloc
$ ./a.out
Segmentation fault: 11
$ lldb ./a.out
...
warning: (arm64) .../jemalloc-libc++-test/a.out(0x0000000100000000) address 0x0000000100000000 maps to more than one section: a.out.__TEXT and a.out.__TEXT
Process 85326 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x0000000000000000
error: memory read failed for 0x0
Target 0: (a.out) stopped.
(lldb) bt
* frame #0: 0x0000000000000000
frame #1: 0x000000010046dbb8 libc++.1.0.dylib`std::__1::__stdinbuf<char>::imbue(std::__1::locale const&) + 52
frame #2: 0x000000010046d41c libc++.1.0.dylib`std::__1::DoIOSInit::DoIOSInit() + 148
frame #3: 0x000000010046e9d4 libc++.1.0.dylib`_GLOBAL__I_000100 + 72
frame #4: 0x000000019e8581d8 dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
...
frame #10: 0x000000019e899904 dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 432
...
frame #16: 0x000000019e854d90 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 304
frame #17: 0x000000019e878984 dyld`dyld4::APIs::runAllInitializersForMain() + 468
frame #18: 0x000000019e83d2d0 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3480
frame #19: 0x000000019e83be18 dyld`start + 1964
You can also see that a C program does not segfault, because (I believe) there is only one libc++.so linked at runtime to satisfy libjemalloc.dylib. And hence there is no funny business, dyld can run the initialisers without failing.
$ echo 'int main() {}' > hello.c
$ clang hello.c -ljemalloc
$ ./a.out
FWIW all the dlopen-tests pass on arm64.
I just realized that, after updating nixpkgs, I'm seeing something similar when my editor tries to start blackd
:
Process: python3.11 [18476]
Path: /Volumes/VOLUME/*/python3.11
Identifier: python3.11
Version: 0
Code Type: X86-64 (Native)
Parent Process: bash [1228]
Responsible: Terminal [951]
User ID: 501
Date/Time: 2023-12-11 12:21:58.122 -0600
OS Version: Mac OS X 10.15.7 (19H2026)
Report Version: 12
Bridge OS Version: 6.6 (19P6064)
...
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [18476]
VM Regions Near 0:
-->
__TEXT 000000010311c000-000000010311d000 [ 4K] r-x/r-x SM=COW /Volumes/VOLUME/*/*.11
Application Specific Information:
/nix/store/x4s0wi6d2z38wa0jb4yf769sl1c0c50z-libcxx-16.0.6/lib/libc++.1.0.dylib
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 ??? 000000000000000000 0 + 0
1 libc++.1.0.dylib 0x0000000104d53ab8 std::__1::__stdinbuf<char>::imbue(std::__1::locale const&) + 40
2 libc++.1.0.dylib 0x0000000104d532af std::__1::DoIOSInit::DoIOSInit() + 143
3 libc++.1.0.dylib 0x0000000104d548bd _GLOBAL__I_000100 + 45
4 dyld 0x000000010391c353 ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 535
5 dyld 0x000000010391c75e ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
6 dyld 0x000000010391717b ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 493
7 dyld 0x00000001039170e6 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 344
8 dyld 0x0000000103915234 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
9 dyld 0x00000001039152d4 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
10 dyld 0x0000000103906ef2 dyld::runInitializers(ImageLoader*) + 82
11 dyld 0x000000010391101b dlopen_internal + 609
12 libdyld.dylib 0x00007fff69c57d8a dlopen + 171
13 libpython3.11.dylib 0x00000001032d30c7 _PyImport_FindSharedFuncptr + 199
14 libpython3.11.dylib 0x000000010329b1a0 _PyImport_LoadDynamicModuleWithSpec + 464
15 libpython3.11.dylib 0x000000010329a941 _imp_create_dynamic + 305
16 libpython3.11.dylib 0x00000001031bd8c1 cfunction_vectorcall_FASTCALL + 193
17 libpython3.11.dylib 0x0000000103266dce _PyEval_EvalFrameDefault + 67838
18 libpython3.11.dylib 0x0000000103256420 _PyEval_Vector + 336
19 libpython3.11.dylib 0x0000000103176b8f object_vacall + 335
20 libpython3.11.dylib 0x00000001031769e7 PyObject_CallMethodObjArgs + 215
21 libpython3.11.dylib 0x0000000103298f3c PyImport_ImportModuleLevelObject + 1404
22 libpython3.11.dylib 0x0000000103251f48 builtin___import__ + 248
23 libpython3.11.dylib 0x00000001031bd951 cfunction_vectorcall_FASTCALL_KEYWORDS + 129
24 libpython3.11.dylib 0x0000000103266dce _PyEval_EvalFrameDefault + 67838
25 libpython3.11.dylib 0x0000000103256420 _PyEval_Vector + 336
26 libpython3.11.dylib 0x0000000103176b8f object_vacall + 335
27 libpython3.11.dylib 0x00000001031769e7 PyObject_CallMethodObjArgs + 215
28 libpython3.11.dylib 0x0000000103298fe2 PyImport_ImportModuleLevelObject + 1570
29 libpython3.11.dylib 0x000000010325fe0e _PyEval_EvalFrameDefault + 39230
30 libpython3.11.dylib 0x000000010325627a PyEval_EvalCode + 202
31 libpython3.11.dylib 0x0000000103252e70 builtin_exec + 832
32 libpython3.11.dylib 0x00000001031bd951 cfunction_vectorcall_FASTCALL_KEYWORDS + 129
33 libpython3.11.dylib 0x0000000103266dce _PyEval_EvalFrameDefault + 67838
34 libpython3.11.dylib 0x0000000103256420 _PyEval_Vector + 336
35 libpython3.11.dylib 0x0000000103176b8f object_vacall + 335
36 libpython3.11.dylib 0x00000001031769e7 PyObject_CallMethodObjArgs + 215
37 libpython3.11.dylib 0x0000000103298f3c PyImport_ImportModuleLevelObject + 1404
38 libpython3.11.dylib 0x000000010325fe0e _PyEval_EvalFrameDefault + 39230
39 libpython3.11.dylib 0x000000010325627a PyEval_EvalCode + 202
40 libpython3.11.dylib 0x0000000103252e70 builtin_exec + 832
41 libpython3.11.dylib 0x00000001031bd951 cfunction_vectorcall_FASTCALL_KEYWORDS + 129
42 libpython3.11.dylib 0x0000000103266dce _PyEval_EvalFrameDefault + 67838
43 libpython3.11.dylib 0x0000000103256420 _PyEval_Vector + 336
44 libpython3.11.dylib 0x0000000103176b8f object_vacall + 335
45 libpython3.11.dylib 0x00000001031769e7 PyObject_CallMethodObjArgs + 215
46 libpython3.11.dylib 0x0000000103298f3c PyImport_ImportModuleLevelObject + 1404
47 libpython3.11.dylib 0x000000010325fe0e _PyEval_EvalFrameDefault + 39230
48 libpython3.11.dylib 0x000000010325627a PyEval_EvalCode + 202
49 libpython3.11.dylib 0x0000000103252e70 builtin_exec + 832
50 libpython3.11.dylib 0x00000001031bd951 cfunction_vectorcall_FASTCALL_KEYWORDS + 129
51 libpython3.11.dylib 0x0000000103266dce _PyEval_EvalFrameDefault + 67838
52 libpython3.11.dylib 0x0000000103256420 _PyEval_Vector + 336
53 libpython3.11.dylib 0x0000000103176b8f object_vacall + 335
54 libpython3.11.dylib 0x00000001031769e7 PyObject_CallMethodObjArgs + 215
55 libpython3.11.dylib 0x0000000103298f3c PyImport_ImportModuleLevelObject + 1404
56 libpython3.11.dylib 0x000000010325fe0e _PyEval_EvalFrameDefault + 39230
57 libpython3.11.dylib 0x000000010325627a PyEval_EvalCode + 202
58 libpython3.11.dylib 0x0000000103252e70 builtin_exec + 832
59 libpython3.11.dylib 0x00000001031bd951 cfunction_vectorcall_FASTCALL_KEYWORDS + 129
60 libpython3.11.dylib 0x0000000103266dce _PyEval_EvalFrameDefault + 67838
61 libpython3.11.dylib 0x0000000103256420 _PyEval_Vector + 336
62 libpython3.11.dylib 0x0000000103176b8f object_vacall + 335
63 libpython3.11.dylib 0x00000001031769e7 PyObject_CallMethodObjArgs + 215
64 libpython3.11.dylib 0x0000000103298f3c PyImport_ImportModuleLevelObject + 1404
65 libpython3.11.dylib 0x000000010325fe0e _PyEval_EvalFrameDefault + 39230
66 libpython3.11.dylib 0x000000010325627a PyEval_EvalCode + 202
67 libpython3.11.dylib 0x00000001032b58d3 run_mod + 147
68 libpython3.11.dylib 0x00000001032b3ef2 _PyRun_SimpleFileObject + 738
69 libpython3.11.dylib 0x00000001032b39eb _PyRun_AnyFileObject + 123
70 libpython3.11.dylib 0x00000001032d3abc Py_RunMain + 2300
71 libpython3.11.dylib 0x00000001032d3f8d pymain_main + 397
72 libpython3.11.dylib 0x00000001032d4009 Py_BytesMain + 57
73 libdyld.dylib 0x00007fff69c6ccc9 start + 1
Thread 0 crashed with X86 Thread State (64-bit):
...
Logical CPU: 2
Error Code: 0x00000014 (no mapping for user instruction read)
Trap Number: 14
...
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This really looks like an impurity bug. I only get this on a 11.7 machine, but not 10.14.6.
I found some interesting discussion in https://trac.macports.org/ticket/62426. The DYLD_INSERT_LIBRARIES
workaround that was mentioned there works for me.
I'm wondering if slapping -DLIBCXX_LIBCPPABI_VERSION=2
, like we did up to libcxx 10, would solve the issue. That would modify the suffix of _LIBCPP_ABI_NAMESPACE
and effectively implement a suggestion from the discussion in https://groups.google.com/g/llvm-dev/c/6ZZ0FjiueDY .
I'm wondering if slapping
-DLIBCXX_LIBCPPABI_VERSION=2
, like we did up to libcxx 10, would solve the issue. That would modify the suffix of_LIBCPP_ABI_NAMESPACE
and effectively implement a suggestion from the discussion in https://groups.google.com/g/llvm-dev/c/6ZZ0FjiueDY .
using a variant of the bug described in https://github.com/NixOS/nixpkgs/issues/269548#issuecomment-1849007022 adding -DLIBCXX_LIBCPPABI_VERSION=2 doesn't appear to have any effect.
this is run on 12.7.1 and the test code is c, linked to llvmPackages_16.libcxx and will dlopen the list of libraries passed in on the command line. it segfaults when dlopening llvmPackages_15.libcxx. when not linked can dlopen all of the libc++ it wants. libcxx versions prior to what is included in clang-15 do not seem to have an issue, though are not run by default in the test below.
can run test using nix-shell and the shell.nix
below.
I'm wondering if slapping
-DLIBCXX_LIBCPPABI_VERSION=2
, like we did up to libcxx 10, would solve the issue.
It looks like this option was removed in https://github.com/llvm/llvm-project/commit/efcee4b06d2f8ee6c79dd893b702f073593d5823. I do wonder if this is merely a coincidence, but GH indicates the commit was first released in libcxx 15.
Then -DLIBCXX_ABI_VERSION=2
or -DLIBCXX_ABI_NAMESPACE=__nix
.
I'm not sure how well this test mirrors reality, but it yielded:
...
-- Installing: /nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6/lib/libc++experimental.a
post-installation fixup
/nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6/lib/libc++.1.0.dylib: fixing dylib
checking for references to /private/tmp/nix-build-libcxx-16.0.6.drv-1/ in /nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6...
patching script interpreter paths in /nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6
stripping (with command strip and flags -S) in /nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6/lib
checking for references to /private/tmp/nix-build-libcxx-16.0.6.drv-1/ in /nix/store/10dv0z5xrfaiybi1d1qwkzix1vcmfrd8-libcxx-16.0.6-dev...
patching script interpreter paths in /nix/store/10dv0z5xrfaiybi1d1qwkzix1vcmfrd8-libcxx-16.0.6-dev
building '/nix/store/w4b32bwimihjhb4q6h999hclzswnj6mg-runtests.drv'...
dyld: calling initializer function 0x7fff66c506b5 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x1055d3770 in /nix/store/api4m8lkvqyj939234fmbgbwwmkp2j7x-libcxx-16.0.6/lib/libc++.1.0.dylib
dyld: calling initializer function 0x7fff66f4a6f8 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x10573c110 in /nix/store/1mk47dp3jgm02gigc5j65lckafai97kr-libcxx-15.0.7/lib/libc++.dylib
dlopen /nix/store/1mk47dp3jgm02gigc5j65lckafai97kr-libcxx-15.0.7/lib/libc++.dylib: succeeded
dyld: calling initializer function 0x105890660 in /nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6/lib/libc++.dylib
dlopen /nix/store/dn60a1f690403ip7fnnpnlk2k4vk16bj-libcxx-16.0.6/lib/libc++.dylib: succeeded
I'm trying to apply a hardcoded version of this (i.e., "-DLIBCXX_ABI_NAMESPACE=__nix16"
) on top of a nixpkgs forked off from nixpkgs-23.11-darwin
@ 473ed42912f3 to see if it can fix the nix segfault I've been seeing. So far I've had to apply it to libcxx and lld. This is on a spare 2013 air, so it'll probably take an age :)
Failed on the latter. These pop on missing symbols while building lld. I thought maybe I had fixed this by adding the namespace to lld as well, but I was just confusing two of three lld builds--one succeeds, one fails, and one is later in the process.
The one that succeeds is first below:
succeeds: $ nix why-depends /nix/store/nkr6afk4xrhyf17v3g6f6d9jx4yfn399-nix-2.18.1.drv /nix/store/q14nwr8bii9n47gx5zm0jg81xaani1k5-lld-16.0.6.drv --extra-experimental-features nix-command
/nix/store/nkr6afk4xrhyf17v3g6f6d9jx4yfn399-nix-2.18.1.drv
└───/nix/store/ncq012mndj4zrs0996162clfp2cysp1n-stdenv-darwin.drv
└───/nix/store/5w0as5w8c6mdxyg73pk4as2pflv23jxp-Libsystem-1238.60.2.drv
└───/nix/store/cjcc12khsgv5a9l8ysix2i5ka8ig6zw1-cctools-llvm-16.0.6-973.0.1.drv
└───/nix/store/vx982ya6v945skbgv02r2h0cyrzqfc12-llvm-binutils-16.0.6.drv
└───/nix/store/q14nwr8bii9n47gx5zm0jg81xaani1k5-lld-16.0.6.drv
fails: $ nix why-depends /nix/store/nkr6afk4xrhyf17v3g6f6d9jx4yfn399-nix-2.18.1.drv /nix/store/w2g253cjvqx72jpnzp3plfmms0y27sv4-lld-16.0.6.drv --extra-experimental-features nix-command
/nix/store/nkr6afk4xrhyf17v3g6f6d9jx4yfn399-nix-2.18.1.drv
└───/nix/store/4j9rd7f5iph2mi19m445p1zndm0xbr59-curl-8.4.0.drv
└───/nix/store/ha5i97gnbaw6fk7nj06ipbg706ndxsvg-libidn2-2.3.4.drv
└───/nix/store/k9ncc6560004520vq0z7kq9q95ql9mf2-bootstrap-stage4-clang-wrapper-16.0.6.drv
└───/nix/store/gh1zg6jgcf1ynv3yvanny7083mcwd1nh-cctools-binutils-darwin-wrapper-16.0.6-973.0.1.drv
└───/nix/store/gh27i2fbyg07hx29hknaykc6yd1z12fq-cctools-binutils-darwin-16.0.6-973.0.1.drv
└───/nix/store/kwdrw3fzlg48x5dx9c66z3j5w56mwkgq-cctools-llvm-16.0.6-973.0.1.drv
└───/nix/store/gdw5ajlqd5gqnhkrbdv2cqhxn13v9p0f-llvm-binutils-16.0.6.drv
└───/nix/store/w2g253cjvqx72jpnzp3plfmms0y27sv4-lld-16.0.6.drv
not run: $ nix why-depends /nix/store/nkr6afk4xrhyf17v3g6f6d9jx4yfn399-nix-2.18.1.drv /nix/store/6yb687yf2b2mwn1s84fwpsnjfd5v9i7j-lld-16.0.6.drv --extra-experimental-features nix-command
/nix/store/nkr6afk4xrhyf17v3g6f6d9jx4yfn399-nix-2.18.1.drv
└───/nix/store/ncq012mndj4zrs0996162clfp2cysp1n-stdenv-darwin.drv
└───/nix/store/6yb687yf2b2mwn1s84fwpsnjfd5v9i7j-lld-16.0.6.drv
@abathur Thanks for investigating! You mean the one that succeeds is the bootrap one, and you can't get to stdenv?
I did have to double-check that I didn't get them switched around, but no--I got the order right. There are actually 3 llds in the build list, but it doesn't reach the third. I'll update the previous post with that one momentarily.
When I run @abathur’s test on macOS 14, this is the output I get. What’s bringing in /usr/lib/libc++.1.dylib
?
dyld[18325]: running initializer 0x18f596590 in /usr/lib/libSystem.B.dylib
dyld[18325]: running initializer 0x10305287c in /nix/store/q6xbm41j3x3sqbachbhxff5mlyn01280-libcxx-16.0.6/lib/libc++.1.0.dylib
dyld[18325]: running initializer 0x102c217c0 in /nix/store/9ybjk0l75sl9jgsgc8yx4s1pcqwnq2jv-libcxx-15.0.7/lib/libc++.1.0.dylib
dlopen /nix/store/9ybjk0l75sl9jgsgc8yx4s1pcqwnq2jv-libcxx-15.0.7/lib/libc++.dylib: succeeded
dyld[18325]: running initializer 0x10315e75c in /nix/store/zb8azf35inr8kwbysv0y960bpn090z4g-libcxx-16.0.6/lib/libc++.1.0.dylib
dlopen /nix/store/zb8azf35inr8kwbysv0y960bpn090z4g-libcxx-16.0.6/lib/libc++.dylib: succeeded
This is what I get on macOS 14. The format is a bit different probably because dyld rewrite that was introduced in macOS 13. It’s mapping the system libc++ but only binding symbols from the nixpkgs libc++.
https://gist.github.com/reckenrode/57e13032286cb71766c92ca6721487c1
This patch lets me build libc++ with LIBCXX_ABI_NAMESPACE=__nix
.
diff --git a/pkgs/development/compilers/llvm/16/libcxx/default.nix b/pkgs/development/compilers/llvm/16/libcxx/default.nix
index 78cd632024cd..c3e7c336c9a3 100644
--- a/pkgs/development/compilers/llvm/16/libcxx/default.nix
+++ b/pkgs/development/compilers/llvm/16/libcxx/default.nix
@@ -74,6 +74,8 @@ stdenv.mkDerivation rec {
"-DLLVM_ENABLE_RUNTIMES=libcxx"
"-DLIBCXX_CXX_ABI=${if headersOnly then "none" else libcxx_cxx_abi_opt}"
] ++ lib.optional (!headersOnly && cxxabi.libName == "c++abi") "-DLIBCXX_CXX_ABI_INCLUDE_PATHS=${cxxabi.dev}/include/c++/v1"
+ # Avoid conflicts with the system libc++ on Darwin.
+ ++ lib.optional stdenv.hostPlatform.isDarwin "-DLIBCXX_ABI_NAMESPACE=__nix"
++ lib.optional (stdenv.hostPlatform.isMusl || stdenv.hostPlatform.isWasi) "-DLIBCXX_HAS_MUSL_LIBC=1"
++ lib.optionals (stdenv.hostPlatform.useLLVM or false) [
"-DLIBCXX_USE_COMPILER_RT=ON"
diff --git a/pkgs/stdenv/darwin/default.nix b/pkgs/stdenv/darwin/default.nix
index c94c56daae1c..3a76adaa1456 100644
--- a/pkgs/stdenv/darwin/default.nix
+++ b/pkgs/stdenv/darwin/default.nix
@@ -870,8 +873,8 @@ in
darwin = super.darwin.overrideScope (selfDarwin: superDarwin: {
inherit (prevStage.darwin)
- Libsystem configd darwin-stubs launchd locale print-reexports rewrite-tbd
- signingUtils sigtool;
+ Libsystem cctools cctools-llvm cctools-port configd darwin-stubs launchd locale
+ print-reexports rewrite-tbd signingUtils sigtool;
# Rewrap binutils so it uses the rebuilt Libsystem.
binutils = superDarwin.binutils.override {
@@ -882,9 +885,6 @@ in
} // {
passthru = { inherit (prevStage.bintools.passthru) isFromBootstrapFiles; };
};
-
- # Avoid building unnecessary Python dependencies due to building LLVM manpages.
- cctools-llvm = superDarwin.cctools-llvm.override { enableManpages = false; };
});
llvmPackages = super.llvmPackages // (
@@ -960,10 +960,10 @@ in
]);
assert lib.all isBuiltByBootstrapFilesCompiler (with prevStage.darwin; [
- locale print-reexports rewrite-tbd sigtool
+ cctools locale print-reexports rewrite-tbd sigtool
]);
assert lib.all isBuiltByNixpkgsCompiler (with prevStage.darwin; [
- binutils-unwrapped cctools libtapi
+ binutils-unwrapped libtapi
]);
assert (! useAppleSDKLibs) -> lib.all isBuiltByBootstrapFilesCompiler (with prevStage.darwin; [ configd ]);
@@ -999,9 +999,12 @@ in
darwin = super.darwin.overrideScope (selfDarwin: superDarwin: {
inherit (prevStage.darwin)
- CF Libsystem binutils binutils-unwrapped cctools cctools-llvm cctools-port configd
+ CF Libsystem binutils binutils-unwrapped configd
darwin-stubs dyld launchd libclosure libdispatch libobjc libtapi locale objc4
postLinkSignHook print-reexports rewrite-tbd signingUtils sigtool;
+
+ # Avoid building unnecessary Python dependencies due to building LLVM manpages.
+ cctools-llvm = superDarwin.cctools-llvm.override { enableManpages = false; };
});
llvmPackages = super.llvmPackages // (
patch works on x86-64 11.7.10 on master @ fa094c6dd42f8e62334a146e463e3e4684d405c0. with no patch, python import requests
segfaults. when using patch, import requests
succeeds.
$ nix-shell -I nixpkgs=./unpatched -p 'python3.withPackages(p: [p.requests])' --run 'python -c "import requests; exit()"'
/private/tmp/nix-shell-50410-0/rc: line 3: 50416 Segmentation fault: 11 python -c "import requests; exit()"
$ nix-shell -I nixpkgs=./patched -p 'python3.withPackages(p: [p.requests])' --run 'python -c "import requests; exit()"'
$
test case from the opening comment of bug report also fixed for libcxx-16-0.6 (note libcxx-15.0.7 is not patched)
***** testing dlopen of libcxx ****
dlopen /nix/store/3vgjjc8dhdazx4bxsamr8765ang66nfx-libcxx-6.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/jyqqw1v1nqwwh804cc3piwxafwz28j8k-libcxx-7.1.0/lib/libc++.dylib: succeeded
dlopen /nix/store/cbjzgibqfikbj4s75126ki2rk8fg3pv9-libcxx-8.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/qpi6yv0x5r1wqhwqdzbkrkx7hyq714ia-libcxx-9.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/gkw121rm91qmad6qw05qqlvbxl1r1qy4-libcxx-10.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/j6whmbi6xdr55gxl4s9qqjqi62kvagn6-libcxx-11.1.0/lib/libc++.dylib: succeeded
dlopen /nix/store/xz4bbl2ifh47vmj6wgjfn0gdx5nq409n-libcxx-12.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/vxxkdhhdqm5gk4lpxvkjpqbw28z9fgi5-libcxx-13.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/4kg7hq00vbn9d8qydypi1rwfvj834d0p-libcxx-14.0.6/lib/libc++.dylib: succeeded
/nix/store/yl4b1fqyrp6yv9514q17n7gc0jd39dh6-runtests/bin/runtests: line 3: 94133 Segmentation fault: 11 dltest_$x
dltest_15 failed with 139
dlopen /nix/store/jhzsigl1fnq7bs9xa2yqgydndnbmd1kz-libcxx-16.0.6/lib/libc++.dylib: succeeded
***** testing dlopen of libcxx when linked to executable *****
dlopen /nix/store/3vgjjc8dhdazx4bxsamr8765ang66nfx-libcxx-6.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/jyqqw1v1nqwwh804cc3piwxafwz28j8k-libcxx-7.1.0/lib/libc++.dylib: succeeded
dlopen /nix/store/cbjzgibqfikbj4s75126ki2rk8fg3pv9-libcxx-8.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/qpi6yv0x5r1wqhwqdzbkrkx7hyq714ia-libcxx-9.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/gkw121rm91qmad6qw05qqlvbxl1r1qy4-libcxx-10.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/j6whmbi6xdr55gxl4s9qqjqi62kvagn6-libcxx-11.1.0/lib/libc++.dylib: succeeded
dlopen /nix/store/xz4bbl2ifh47vmj6wgjfn0gdx5nq409n-libcxx-12.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/vxxkdhhdqm5gk4lpxvkjpqbw28z9fgi5-libcxx-13.0.1/lib/libc++.dylib: succeeded
dlopen /nix/store/4kg7hq00vbn9d8qydypi1rwfvj834d0p-libcxx-14.0.6/lib/libc++.dylib: succeeded
dlopen /nix/store/v1cr8zqni615r6fr1j6kgwxrjpmqsm0m-libcxx-15.0.7/lib/libc++.dylib: succeeded
dlopen /nix/store/jhzsigl1fnq7bs9xa2yqgydndnbmd1kz-libcxx-16.0.6/lib/libc++.dylib: succeeded
libcxx-15
and libcxx-16
are no longer defining LIBCXX_OSX_REEXPORT_LIBCXXABI_SYMBOLS=ON
due to a name change LIBCXX_CXX_ABI=system-libcxxabi
rather than libcxxabi
in default.nix. This changes how some symbols are defined.
if (APPLE AND LIBCXX_CXX_ABI STREQUAL "libcxxabi"
AND NOT DEFINED LIBCXX_OSX_REEXPORT_LIBCXXABI_SYMBOLS
AND NOT LIBCXX_STATICALLY_LINK_ABI_IN_SHARED_LIBRARY)
set(LIBCXX_OSX_REEXPORT_LIBCXXABI_SYMBOLS ON)
endif()
if (LIBCXX_OSX_REEXPORT_LIBCXXABI_SYMBOLS)
target_link_libraries(cxx_shared PRIVATE
"-Wl,-unexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++unexp.exp"
"-Wl,-reexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++abi.exp"
"-Wl,-force_symbols_not_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/notweak.exp"
"-Wl,-force_symbols_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/weak.exp")
target_link_libraries(cxx_shared PRIVATE $<TARGET_NAME_IF_EXISTS:cxxabi-reexports>)
endif()
forcing -DLIBCXX_OSX_REEXPORT_LIBCXXABI_SYMBOLS=ON
allows x86-64 11.7.10 to pass the dlopen tests and can build nix and run nix-shell -p ponysay --dry-run
also passes without segfaulting.
Nice find!
The LIBCXX_CXX_ABI
change is probably due to https://github.com/llvm/llvm-project/commit/a80e65e00ada7a9c16acf17a5fd40b4f12ced3a8. Looking at the linked review, the assumption appears to be that Darwin will link against the in-tree libc++abi, but nixpkgs is building and using a “system” one.
I messed around with the link options a little.
target_link_libraries(cxx_shared PRIVATE
"-Wl,-unexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++unexp.exp"
"-Wl,-reexported_symbols_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/libc++abi.exp"
"-Wl,-force_symbols_not_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/notweak.exp"
"-Wl,-force_symbols_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/weak.exp")
and the only one needed to prevent the dlopen crashes is
"-Wl,-force_symbols_not_weak_list,${CMAKE_CURRENT_SOURCE_DIR}/../lib/notweak.exp"
-- where lib/notweak.exp
is:
# Remove the weak-def bit from these external symbols
__ZT*
__ZN*
__ZS*
though not sure how the symbols get marked weak in the first place. there is a macro _LIBCPP_WEAK
that expands to __attribute__((weak))
but removing does not resolve the segfaults and it is only marking lib/new.cpp
Fixed: likely going to take some time to make it from staging to master https://nixpk.gs/pr-tracker.html?pr=278945 and release-23.11 https://nixpk.gs/pr-tracker.html?pr=280518
Describe the bug
libcxx is a c++ standard library used on x64 darwin and when it is loaded by dlopen or as a result of a dlopen eg:
import requests
on python the program segfaults in the initialization of libcxx.older versions of libcxx [6-14] dlopen fine. when libcxx is linked into main executable dlopen is also fine.
the macOS system was created by the steps outlined https://github.com/kholia/OSX-KVM selecting the
big sur
install media and run under qemu on linux.note that a vm setup using the
monterey
install media does not have this issue.Steps To Reproduce
just need to dlopen the lib. see below for a nix-shell script which builds for libcxx[6, 16].
Expected behavior
libcxx loads into the address space of the program and the initialization routines succeed.
Additional context
dltest_15_2023-11-23-183122_odel.crash.txt dltest_16_2023-11-23-183122_odel.crash.txt python3.11_2023-11-22-002452_odel.crash.txt
Notify maintainers
@dtzWill @Ericson2314 @lovek323 @primeos @alyssais @RaitoBezarius @rrbutani @sternenseemann
Metadata
Priorities
Add a :+1: reaction to issues you find important.