[Python][Mac][Homebrew][Installation issue] Cannot install `pyarrow` if `apache-arrow` is installed with homebrew

[gbelouze]

Describe the bug, including details regarding any error messages, version, and platform.

Not sure where the issue is coming from. Don't think this matters but this is tested inside a conda environment. I also reproduced inside a venv managed by uv.

Issue

$ pip install pyarrow
# no problem
$ python -c "import pyarrow"
# no problem

BUT

$ brew install apache-arrow
$ pip install pyarrow
# no problem
$ python -c "import pyarrow"
❯ python -c "import pyarrow"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/Caskroom/miniconda/base/envs/test/lib/python3.12/site-packages/pyarrow/__init__.py", line 65, in <module>
    import pyarrow.lib as _lib
ImportError: dlopen(/opt/homebrew/Caskroom/miniconda/base/envs/test/lib/python3.12/site-packages/pyarrow/lib.cpython-312-darwin.so, 0x0002): Symbol not found: __ZN6google8protobuf11safe_strtobENS0_20stringpiece_internal11StringPieceEPb
  Referenced from: <99FF4B65-BC25-31C0-9721-25CEAF38A760> /opt/homebrew/Caskroom/miniconda/base/envs/test/lib/python3.12/site-packages/pyarrow/libarrow_substrait.1700.dylib
  Expected in:     <CD906312-0048-3845-816B-F382564EB23E> /opt/homebrew/Cellar/apache-arrow/17.0.0_2/lib/libarrow.1700.0.0.dylib

$ brew uninstall apach-arrow
$ python -c "import pyarrow"
# no problem

On MacOS Sonoma 14.6 (M3 Silicon). Python version is 3.12 and pyarrow version is 17.0.0. Any help is appreciated !

Component(s)

Python

Edit: include the full stack trace of python -c "import pyarrow"

[kou]

Could you show otool -L /opt/homebrew/Caskroom/miniconda/base/envs/test/lib/python3.12/site-packages/pyarrow/libarrow_substrait.1700.dylib?

[gbelouze]

Thanks for the quick response. I get

/opt/homebrew/Caskroom/miniconda/base/envs/test/lib/python3.12/site-packages/pyarrow/libarrow_substrait.1700.dylib:
        @rpath/libarrow_substrait.1700.dylib (compatibility version 1700.0.0, current version 1700.0.0)
        @rpath/libarrow_dataset.1700.dylib (compatibility version 1700.0.0, current version 1700.0.0)
        @rpath/libparquet.1700.dylib (compatibility version 1700.0.0, current version 1700.0.0)
        @rpath/libarrow_acero.1700.dylib (compatibility version 1700.0.0, current version 1700.0.0)
        @rpath/libarrow.1700.dylib (compatibility version 1700.0.0, current version 1700.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1600.151.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

[kou]

Could you show otool -l $(which python)? I want to know @rpath in this context. LC_RPATH entry in the otool -l output will show it.

[gbelouze]

❯ otool -l $(which python3)

otool.txt (sorry for the multiple edits)

[amoeba]

I wasn't able to reproduce this with a fresh Sonoma VM. @gbelouze just so we're sure on your exact steps, could you upload a complete and un-edited log of the commands you're running starting from conda environment creation?

[kou]

Thanks.

/opt/homebrew/Caskroom/miniconda/base/envs/test/bin/python3:
...
Load command 17
          cmd LC_RPATH
      cmdsize 40
         path @loader_path/../lib/ (offset 12)

@rpath in this context should be /opt/homebrew/Caskroom/miniconda/base/envs/test/lib.

Hmm. Both of libarrow*.dylib from PyArrow wheel and apache-arrow formula don't exist in there...

Could you use python not python3 because python -c "import pyarrow" uses python not python3? otool -l $(which python)?

[gbelouze]

I have python alias to python3. I'll produce a complete a log momentarily.

[gbelouze]

Solved the issue. I recently added this line to my .zshrc

export DYLD_LIBRARY_PATH="$(brew --prefix)/lib:$DYLD_LIBRARY_PATH"

as was advised in the install instructions of https://github.com/3rd/image.nvim Removing the line fixed the issue. Looking back, it seemed incredibly obvious.

I'll leave this issue open in case you need further info - but feel free to close it otherwise. I appreciate all the help even if in the end the issue was purely on my side.

[amoeba]

Thanks for reporting back @gbelouze. That's not normally a step that would be required so I think something may be misconfigured with your environment. A full log may still be useful to help troubleshoot your setup.

[kou]

I think that we can close this.

If image.nvim works well without the DYLD_LIBRARY_PATH change, could you report it (DYLD_LIBRARY_PATH change is needless with recent Homebrew) to image.nvim?