Error: Library not loaded: @rpath/libunwind.1.dylib on macOS M2

[meftunca]

Description: I encountered an error while running the project on my MacBook Pro M2 (Apple Silicon). The error message is as follows:

dyld[70274]: Library not loaded: @rpath/libunwind.1.dylib
Referenced from: <BE8B7E7B-8446-3C43-BA2E-B6EAA7E8D61F> /private/var/tmp/_bazel_*****/bebcbb61406245ffe61733b8c2f2ccde/execroot/sondb/bazel-out/darwin_arm64-dbg/bin/src/sondb
Reason: tried: '/usr/local/lib/libunwind.1.dylib' (no such file), '/usr/lib/libunwind.1.dylib' (no such file, not in dyld cache)
make: *** [debug-run] Error 255

It seems to be related to a missing or incorrect library (libunwind) on macOS M2. I believe this issue is specific to the project configuration.

Environment:

macOS version:13.5.2 ventura (m2 max) Project version:0.10.3 LLVM: 15.0.6/15.0.7/16.0.0

I would appreciate any help in resolving this issue. Thank you!

[mickeyh]

I have experienced the same error on an M2 Mac Mini. It occurs because:

1. On the ARM-based Macs bazel-toolchain adds the following compiler flag: -L/private/var/tmp/<path to some Bazel directory>/external/llvm_toolchain_llvm/lib and /private/var/tmp/<path to some Bazel directory>/external/llvm_toolchain_llvm/lib contains a libunwind dylib.

2. As a result, the produced binary expects libunwind.1.dylib to be available at runtime:

$ otool -l <my test binary>
...
Load command 18
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libunwind.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1.0.0
...

3. But on Mac OS, libunwind is provided as a part of libSystem (libunwind isn't present as an independent dylib) and so the binary fails at runtime.

If I remove -L/private/var/tmp/<path to some Bazel directory>/external/llvm_toolchain_llvm/lib from the compiler arguments and compile manually, I no longer experience this crash (and can confirm with otool that libunwind is no longer loaded by the resulting binary).

I'm not sure the correct way to fix this though. I noticed that for the x64 Mac OS platform -L/private/var/tmp/<path to some Bazel directory>/external/llvm_toolchain_llvm/lib isn't added to the compilation command. Perhaps it's safe to remove this for AMD64 Mac platform as well? Or, maybe libunwind shouldn't be included in the LLVM libs made available at compile time (at least if both Bazel execution and target platforms are AMD64 Mac OS)?

[siddharthab]

Thank you for investigating this. It saves me a lot of time. Are you using absolute paths by any chance?

We already have some logic around this issue in https://github.com/grailbio/bazel-toolchain/blob/91abcad9e44ea47cff7b95962b7b62fc2e885dc5/toolchain/BUILD.llvm_repo#L71, but I think it will not work with absolute paths. I will try to reproduce the issue and fix.

[mickeyh]

I was indeed using absolute paths. I can confirm that when I stop using them the issue disappears.

Thanks for the super fast response!

[siddharthab]

I was unable to reproduce the issue with or without absolute paths on my M2 Mac Mini. The primary things different from the original report were that I tried with macOS 13.4.2 and 14.0, and that I have XCode Command Line Tools installed. The linker command for me has -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -L<....>/external/llvm_toolchain_llvm/lib.

I don't think we can take out the external/llvm_toolchain_llvm/lib directory completely from the command line. And I don't want to make very disruptive changes without understanding the issue a little more. I will stop work on this issue now pending a reproducible test case and/or more information.

[meftunca]

Is there a quick solution I can apply?

[nolen777]

In case it's helpful as a repro case, I've created https://github.com/nolen777/grail-test/ with steps in the README.

I can build and run a Hello, World program just fine on my M1 Mac Studio, but when I include rules_proto, the built protoc binary fails with the libunwind error.

[siddharthab]

Thanks, but I can not reproduce even with your repro repo.

Can you add a direct dependency for protobuf with the line bazel_dep(name = "protobuf", version = "21.7") in MODULE.bazel and then run bazel run --linkopt=-v --linkopt=-Wl,-v @protobuf//:protoc and post here what you get from Bazel for the link step of protoc? And also, the output of otool -l bazel-bin/external/protobuf~21.7/protoc.

If directly running protoc is working for you, then maybe something is different for the transitions.

I suspect this is something rare in your setup, otherwise more people would have reported this basic issue by now.

[nolen777]

Thanks so much for investigating this!

Building protoc directly with your command:

INFO: From Linking external/protobuf~21.7/protoc:
clang version 16.0.0
Target: aarch64-apple-macosx
Thread model: posix
InstalledDir: external/toolchains_llvm~0.10.3~llvm~llvm_toolchain_llvm/bin
 "/usr/bin/ld" -demangle -lto_library external/toolchains_llvm~0.10.3~llvm~llvm_toolchain_llvm/lib/libLTO.dylib -dynamic -arch arm64 -headerpad_max_install_names -platform_version macos 13.0.0 13.0.0 -syslibroot /Applications/Xcode15.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk -o bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/protoc -L/Applications/Xcode15.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/lib -Lexternal/toolchains_llvm~0.10.3~llvm~llvm_toolchain_llvm/lib bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/_objs/protoc/main.pic.o bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/libprotoc_lib.a bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/libprotobuf.a bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/libprotobuf_lite.a bazel-out/darwin_arm64-dbg/bin/external/zlib~1.2.13/libzlib.a -lm -lc++ -lc++abi -lpthread -lm -lpthread -lm -lpthread -lm -lpthread -lm -v -framework Foundation -lobjc -lSystem external/toolchains_llvm~0.10.3~llvm~llvm_toolchain_llvm/lib/clang/16/lib/darwin/libclang_rt.osx.a
@(#)PROGRAM:ld  PROJECT:dyld-1015.7
BUILD 18:48:48 Aug 22 2023
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
will use ld-classic for: armv6 armv7 armv7s arm64_32 i386 armv6m armv7k armv7m armv7em
LTO support using: LLVM version 16.0.0 (static support for 29, runtime is 29)
TAPI support using: Apple TAPI version 15.0.0 (tapi-1500.0.12.3)
Library search paths:
    /Applications/Xcode15.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk/usr/lib
    external/toolchains_llvm~0.10.3~llvm~llvm_toolchain_llvm/lib
Framework search paths:
ld: warning: ignoring duplicate libraries: '-lm', '-lpthread'
Target @protobuf~21.7//:protoc up-to-date:
  bazel-bin/external/protobuf~21.7/protoc
INFO: Elapsed time: 42.582s, Critical Path: 5.99s
INFO: 197 processes: 9 internal, 188 local.
INFO: Build completed successfully, 197 total actions
INFO: Running command line: bazel-bin/external/protobuf~21.7/protoc
dyld[68176]: Library not loaded: @rpath/libunwind.1.dylib
  Referenced from: <B8F46DCB-2D2D-3758-AB98-E8540FE2BADC> /private/var/tmp/_bazel_dancrosby/f0d93c3f9e9ba2c50b7eea59373de0a3/execroot/_main/bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/protoc
  Reason: tried: '/usr/local/lib/libunwind.1.dylib' (no such file), '/usr/lib/libunwind.1.dylib' (no such file, not in dyld cache)
zsh: abort      bazel run --linkopt=-v --linkopt=-Wl,-v @protobuf//:protoc

otool output:

bazel-bin/external/protobuf~21.7/protoc:
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __PAGEZERO
   vmaddr 0x0000000000000000
   vmsize 0x0000000100000000
  fileoff 0
 filesize 0
  maxprot 0x00000000
 initprot 0x00000000
   nsects 0
    flags 0x0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __TEXT
   vmaddr 0x0000000100000000
   vmsize 0x0000000000480000
  fileoff 0
 filesize 4718592
  maxprot 0x00000005
 initprot 0x00000005
   nsects 7
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000100002da0
      size 0x00000000003bed44
    offset 11680
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x00000001003c1ae4
      size 0x0000000000010ae8
    offset 3939044
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 12 (size of stubs)
Section
  sectname __init_offsets
   segname __TEXT
      addr 0x00000001003d25cc
      size 0x000000000000002c
    offset 4007372
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000016
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __TEXT
      addr 0x00000001003d25f8
      size 0x000000000002ee98
    offset 4007416
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __cstring
   segname __TEXT
      addr 0x0000000100401490
      size 0x000000000005bd70
    offset 4199568
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x000000010045d200
      size 0x0000000000009c75
    offset 4575744
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x0000000100466e78
      size 0x0000000000019180
    offset 4615800
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 232
  segname __DATA_CONST
   vmaddr 0x0000000100480000
   vmsize 0x0000000000018000
  fileoff 4718592
 filesize 98304
  maxprot 0x00000003
 initprot 0x00000003
   nsects 2
    flags 0x10
Section
  sectname __got
   segname __DATA_CONST
      addr 0x0000000100480000
      size 0x000000000000b4c0
    offset 4718592
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 5694 (index into indirect symbol table)
 reserved2 0
Section
  sectname __const
   segname __DATA_CONST
      addr 0x000000010048b4c0
      size 0x0000000000009660
    offset 4764864
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 3
      cmd LC_SEGMENT_64
  cmdsize 312
  segname __DATA
   vmaddr 0x0000000100498000
   vmsize 0x0000000000004000
  fileoff 4816896
 filesize 16384
  maxprot 0x00000003
 initprot 0x00000003
   nsects 3
    flags 0x0
Section
  sectname __data
   segname __DATA
      addr 0x0000000100498000
      size 0x0000000000000e48
    offset 4816896
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x0000000100498e48
      size 0x000000000000068a
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __common
   segname __DATA
      addr 0x0000000100499500
      size 0x000000000000009a
    offset 0
     align 2^6 (64)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 4
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x000000010049c000
   vmsize 0x0000000000854000
  fileoff 4833280
 filesize 8719592
  maxprot 0x00000001
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 5
      cmd LC_DYLD_CHAINED_FIXUPS
  cmdsize 16
  dataoff 4833280
 datasize 692928
Load command 6
      cmd LC_DYLD_EXPORTS_TRIE
  cmdsize 16
  dataoff 5526208
 datasize 588616
Load command 7
     cmd LC_SYMTAB
 cmdsize 24
  symoff 6152496
   nsyms 167551
  stroff 8879224
 strsize 4568440
Load command 8
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 156741
     iextdefsym 156741
     nextdefsym 10592
      iundefsym 167333
      nundefsym 218
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 8833312
  nindirectsyms 11478
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 9
          cmd LC_LOAD_DYLINKER
      cmdsize 32
         name /usr/lib/dyld (offset 12)
Load command 10
     cmd LC_UUID
 cmdsize 24
    uuid B8F46DCB-2D2D-3758-AB98-E8540FE2BADC
Load command 11
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 13.0
      sdk 13.0
   ntools 1
     tool 3
  version 1015.7
Load command 12
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 13
       cmd LC_MAIN
   cmdsize 24
  entryoff 14248
 stacksize 0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1336.0.0
compatibility version 1.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1600.151.0
compatibility version 1.0.0
Load command 16
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libc++abi.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1600.151.0
compatibility version 1.0.0
Load command 17
          cmd LC_LOAD_DYLIB
      cmdsize 96
         name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 2048.1.255
compatibility version 300.0.0
Load command 18
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libobjc.A.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 228.0.0
compatibility version 1.0.0
Load command 19
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name @rpath/libunwind.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1.0.0
compatibility version 1.0.0
Load command 20
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 6114824
 datasize 37672
Load command 21
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 6152496
 datasize 0
Load command 22
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 13447664
 datasize 105208

I'll try to do a bit more experimenting tonight -- might possibly be something in my user .bazelrc or in my tools installation, maybe.

[nolen777]

Actually, I did the quick experiment now. Moving away my ~/.bazelrc file makes the problem go away. I'll try to isolate specifically what is causing the issue and post more info tonight.

[nolen777]

OK: I can reproduce this error with --compilation_mode=dbg:

dancrosby@Dans-Mac-Studio grail-test % bazel run --compilation_mode=dbg --linkopt=-v --linkopt=-Wl,-v @protobuf//:protoc
INFO: Invocation ID: b3b6a338-6ba8-4d28-8154-4bdffec4cb3a
INFO: Analyzed target @protobuf~21.7//:protoc (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target @protobuf~21.7//:protoc up-to-date:
  bazel-bin/external/protobuf~21.7/protoc
INFO: Elapsed time: 0.086s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/protobuf~21.7/protoc
dyld[15365]: Library not loaded: @rpath/libunwind.1.dylib
  Referenced from: <B8F46DCB-2D2D-3758-AB98-E8540FE2BADC> /private/var/tmp/_bazel_dancrosby/f0d93c3f9e9ba2c50b7eea59373de0a3/execroot/_main/bazel-out/darwin_arm64-dbg/bin/external/protobuf~21.7/protoc
  Reason: tried: '/usr/local/lib/libunwind.1.dylib' (no such file), '/usr/lib/libunwind.1.dylib' (no such file, not in dyld cache)
zsh: abort      bazel run --compilation_mode=dbg --linkopt=-v --linkopt=-Wl,-v 
dancrosby@Dans-Mac-Studio grail-test % bazel run --linkopt=-v --linkopt=-Wl,-v @protobuf//:protoc                       
INFO: Invocation ID: ead9311e-3f0a-4c85-9096-1b8a61f845b6
INFO: Build option --compilation_mode has changed, discarding analysis cache.
INFO: Analyzed target @protobuf~21.7//:protoc (0 packages loaded, 2446 targets configured).
INFO: Found 1 target...
Target @protobuf~21.7//:protoc up-to-date:
  bazel-bin/external/protobuf~21.7/protoc
INFO: Elapsed time: 0.176s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/external/protobuf~21.7/protoc
Usage: /private/var/tmp/_bazel_dancrosby/f0d93c3f9e9ba2c50b7eea59373de0a3/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/external/protobuf~21.7/protoc [OPTION] PROTO_FILES
Parse PROTO_FILES and generate output based on the options given:
<<snip>>

That's sufficient to unblock me, I think -- thanks for your help! -- but note that I can build with that flag without bazel-toolchain.

[siddharthab]

Thanks for confirming. I am unable to reproduce this even with -c dbg. It looks like the original report was also using debug mode.

LLVM provides many handles to control the linking of libunwind. Maybe one of those can be useful as a workaround for people affected by it.

[nolen777]

ahhh, sorry for the noise here -- I think -c dbg may have been a red herring -- the real cause, I think, was --spawn-strategy=local. I got it wrong because changing spawn_strategy does not force a rebuild, but changing the -c flag does.

If I do a bazel clean and then bazel run --linkopt=-Wl,-v @protobuf//:protoc in my test repo, it consistently works. If I bazel clean and then bazel run --spawn_strategy=local --linkopt=-Wl,-v @protobuf//:protoc, it consistently hits the libunwind error.

I remember needing spawn_strategy=standalone or =local in the past, but hopefully I can get by without it now.

[siddharthab]

Ah yes, local spawn strategy will have the same issue that using absolute paths has. I can maybe look into providing an option to completely hide the lib directory because it's not always needed. Closing this issue now.

[ruipserra]

I hit this same error when building Sorbet from source (see my issue here). This happened on an M1 Macbook Pro, and also on an Intel-based Macbook Pro running macOS 13.6.1.

Sorbet's config uses absolute paths (link to code). I disabled that option and was able to compile without issues.

--spawn_stragegy=local is also used in a few configs (example), but I didn't have to change those to get around the dyld error.

[siddharthab]

Thanks for reporting. In the linked issue, the Sorbet team says that only some team members have this issue. Maybe it is because the others have libunwind locally installed to /usr/lib or equivalent. Did not really read the whole thread and I have lost some of the context to this issue.

[marknuzz]

I have the same error that @ruipserra had at https://github.com/sorbet/sorbet/issues/7528, when building Sorbet.

Changing absolute_paths to False didn't fix it, and neither did changing spawn_strategy to standalone. I've also tried symlinking the /usr/local/Cellar/llvm/17.0.6_1/lib/libunwind.1.dylib to /usr/local/lib/ and sudo ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/* /usr/local/include/.

None of these worked. I'm out of ideas.

Using intel macOS 14.3

Ran bazel clean --expunge after trying every possible fix Command: ./bazel build //main:sorbet --config=dbg --verbose_failures