MacOS install error: Library not loaded: @rpath/libc++.1.dylib

koaning commented 4 years ago

🐛 Bug

Install on MacOS fails with pip.

To Reproduce

(venv) ➜  whatlies git:(master) ✗ python -m pip install torch
(venv) ➜  whatlies git:(master) ✗ python                            
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/vincent/Development/whatlies/venv/lib/python3.7/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: dlopen(/Users/vincent/Development/whatlies/venv/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so, 9): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Users/vincent/Development/whatlies/venv/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so
  Reason: image not found

Google suggested doing this;

(venv) ➜  whatlies git:(master) ✗ brew install libomp

That unfortunately did not work.

Expected behavior

No error.

Environment

(venv) ➜  whatlies git:(master) ✗ python collect_env.py 
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Mac OSX 10.15.3
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.7
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch==1.3.1
[pip3] torchvision==0.5.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @bdhirsh @seemethere @malfet @walterddr

koaning commented 4 years ago

Note that this error also occurs on 1.4.0.

(venv) ➜  whatlies git:(master) ✗ python collect_env.py      
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A

OS: Mac OSX 10.15.3
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.7
Is CUDA available: N/A
CUDA runtime version: Could not collect
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] Could not collect
(venv) ➜  whatlies git:(master) ✗ import python
zsh: command not found: import
(venv) ➜  whatlies git:(master) ✗ python
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/vincent/Development/whatlies/venv/lib/python3.7/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: dlopen(/Users/vincent/Development/whatlies/venv/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so, 9): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: /Users/vincent/Development/whatlies/venv/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so
  Reason: image not found

ezyang commented 4 years ago

Does conda work for you?

soundmartell commented 4 years ago

lib/python3.7/site-packages/torch/init.py", line 82, in from torch._C import *

koaning commented 4 years ago

@ezyang I typically don't use conda and I haven't tried it. The docs suggest it is not needed though.

soundmartell commented 4 years ago

Ok, I solved the issue, I don't know if it has any unwanted impact in any way but now Torch is working. Please, note I am using Mac Pycharm with a virtual environment so... I copied: libc++.1.dylib from usr/lib and I pace it in my environment: venv/lib/python3.7/site-packages/torch/lib

I tested Torch with this script:

from future import print_function import torch x = torch.rand(5, 3) print(x)

It devolved:

tensor([[0.3633, 0.7173, 0.6055], [0.3442, 0.6892, 0.2950], [0.3909, 0.4718, 0.0202], [0.1055, 0.8912, 0.0667], [0.3819, 0.4413, 0.1544]])

I hope it work for others. I have been 3 days trying to find solutions in Google but not any solution found it.

kaiwencui commented 4 years ago

Ok, I solved the issue, I don't know if it has any unwanted impact in any way but now Torch is working. Please, note I am using Mac Pycharm with a virtual environment so... I copied: libc++.1.dylib from usr/lib and I pace it in my environment: venv/lib/python3.7/site-packages/torch/lib

I tested Torch with this script:

from future import print_function import torch x = torch.rand(5, 3) print(x)

It devolved:

tensor([[0.3633, 0.7173, 0.6055], [0.3442, 0.6892, 0.2950], [0.3909, 0.4718, 0.0202], [0.1055, 0.8912, 0.0667], [0.3819, 0.4413, 0.1544]])

I hope it work for others. I have been 3 days trying to find solutions in Google but not any solution found it.

Can confirm this method works on macOS running on Python 3.8 in venv

koaning commented 4 years ago

Running this works on my side too.

 cp /usr/lib/libc++.1.dylib venv/lib/python3.7/site-packages/torch/lib

Seems like something is not properly packaged along?

ezyang commented 4 years ago

Yes, it sounds like the rpath for libc++ is incorrectly set on the wheel packages. We should not be rpath'ing it at all.

mattip commented 4 years ago

maybe connected to gh-35418, where rpaths to cuda libraries are suspected of causing problems

puthurr commented 3 years ago

Hi All

Have the same issue with torchtext though (torch alone is fine)

Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) [Clang 6.0 (clang-600.0.57)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import torchtext Traceback (most recent call last): File "", line 1, in File "/Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torchtext/init.py", line 42, in _init_extension() File "/Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torchtext/init.py", line 38, in _init_extension torch.ops.load_library(ext_specs.origin) File "/Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torch/_ops.py", line 105, in load_library ctypes.CDLL(path) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ctypes/init.py", line 364, in init self._handle = _dlopen(self._name, mode) OSError: dlopen(/Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torchtext/_torchtext.so, 6): Library not loaded: @rpath/libc++.1.dylib Referenced from: /Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torchtext/_torchtext.so Reason: image not found

Tried the workaround but didn't work.

LDD equivalent shows the following _otool -L /Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torchtext/_torchtext.so /Users/nicolas/projects/PyTorch/venv/lib/python3.7/site-packages/torchtext/_torchtext.so: @rpath/libc10.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libtorch.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libtorch_cpu.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libtorchpython.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

malfet commented 3 years ago

@puthurr what versions of PyTorch and TorchText are you trying to install?

puthurr commented 3 years ago

Just installing the latest available torch in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.6.0) torchvision in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (0.7.0) torchtext in ./venv/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.7.0)

wenqinglim commented 3 years ago

@puthurr I ran the solution suggested above cp /usr/lib/libc++.1.dylib venv/lib/python3.7/site-packages/torch/lib and also still faced the same issue as you described. My Pytorch was also working, but TorchText was not. I was using:

Python 3.7.2
torchtext==0.7.0
torch==1.4.0

I managed to resolve it by upgrading to torch==1.6.0. I see above that you're already on Torch 1.6.0, but you might want to do a double check on your other library versions and their compatibility.

puthurr commented 3 years ago

@malfet @wenqinglim
I solved it by changing the rpath in the library _torchtext.so using the following command (Mac OS X) install_name_tool -change @rpath/libc++.1.dylib /usr/lib/libc++.1.dylib _torchtext.so Now the path is absolute. otool -L _torchtext.so _torchtext.so: @rpath/libc10.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libtorch.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libtorch_cpu.dylib (compatibility version 0.0.0, current version 0.0.0) @rpath/libtorch_python.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

thus importing torchtext works fine now. Not ideal but working.

jqiu2 commented 3 years ago

I have the same issue with torch audio, I ran the command above and face the same issues.

PearlW1 commented 3 years ago

brew install libomp

Don't run "brew install libomp", it destroyed my environment , then i create a new environment, install other configuration。

jrmh96 commented 3 years ago

I am also having trouble with some of the dependencies on Mac Big Sur, specifically from pytorch3d import _C

yields

Referenced from: venv/lib/python3.8/site-packages/pytorch3d/_C.cpython-38-darwin.so Reason: Incompatible library version: _C.cpython-38-darwin.so requires version 1.0.0 or later, but libc10.dylib provides version 0.0.0

This is after changing the rpath using install_name_tool as @puthurr suggested

Environment: torch==1.9.0 pytorch3d==1.3.0

espoirMur commented 2 years ago

@puthurr I ran the solution suggested above cp /usr/lib/libc++.1.dylib venv/lib/python3.7/site-packages/torch/lib and also still faced the same issue as you described. My Pytorch was also working, but TorchText was not. I was using:
Python 3.7.2
torchtext==0.7.0
torch==1.4.0
I managed to resolve it by upgrading to torch==1.6.0. I see above that you're already on Torch 1.6.0, but you might want to do a double check on your other library versions and their compatibility.

upgrading to torch 1.6.0 solved the issue for me.

jskye commented 2 years ago

i have basically the same problem but the library libshm.dylib is missing.

dlopen(/usr/local/lib/python3.9/site-packages/torch/_C.cpython-39-darwin.so, 2): Library not loaded: @loader_path/libshm.dylib
  Referenced from: /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_python.dylib
  Reason: image not found

all answers i found to fix that say to install libomp, which i reinstalled but it didnt fix the issue, though for others that seemed to work in previous years.

i cant find the library libshm in any lib folders on my machine. Im wondering if its included somewhere in libomp and how to point pytorch to it, or if it can be downloaded from somewhere else?

i have the current torch version installed 1.11.0 with Python 3.9 on Mac Big Sur.

Found the libshm library required for sharedmemory alloc on linux machines. Does PyTorch require a bugfix to include this dylib? Or is there any other workaround? @malfet

malfet commented 2 years ago

@jskye Can you please run collect_env and post results here? (Or better open a separate issue)

jskye commented 2 years ago

I received this error when installing DALLE2. i think the problem was that the dream lib it relies on big sleep library which requires CUDA that i dont have as my mac has an older onboard Intel chip. big sleep uses CUDA.. I think the issue pertaining to the above libshm may have been due to some mismatch as i think i had pytorch version 12 installed possibly from anaconda and i also had torchaudio and torchvideo that required the older version 11. In any case after resolving that mismatch i get this CUDA error again. Indeed some other experimental art AIs on google collab required me to switch over to use the CPU to use.

dream 'sharing a sunset at the summit of mount everest with my dog'
Traceback (most recent call last):
  File "/usr/local/bin/dream", line 5, in <module>
    from big_sleep.cli import main
  File "/usr/local/lib/python3.9/site-packages/big_sleep/__init__.py", line 1, in <module>
    from big_sleep.big_sleep import BigSleep, Imagine
  File "/usr/local/lib/python3.9/site-packages/big_sleep/big_sleep.py", line 26, in <module>
    assert torch.cuda.is_available(), 'CUDA must be available in order to use Big Sleep'

pytorch / pytorch