`import pyg_lib` fails due to `OSError: dlopen ... (no such file, not in dyld cache)` on macOS

semihcanturk commented 1 year ago

🐛 Describe the bug

I'm unable to get pyg-lib to import on my Mac. It seems to affect both Intel and M1 Macs (I'm on an Intel, but an M1 user reported the same issue: https://github.com/pyg-team/pyg-lib/issues/211#issuecomment-1467254168). The original issue in the thread may not be relevant, thus I'm opening a new issue.

Minimal example is simply importing pyg_lib:

import pyg_lib

In my case, I was importing torch_geometric.graphgym.config, resulting in the following stack trace:

  File "/Users/semo/PycharmProjects/pretrained-pe/graphgym/act/example.py", line 3, in <module>
    from torch_geometric.graphgym.config import cfg
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_geometric/__init__.py", line 4, in <module>
    import torch_geometric.data
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_geometric/data/__init__.py", line 1, in <module>
    from .data import Data
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_geometric/data/data.py", line 20, in <module>
    from torch_sparse import SparseTensor
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/__init__.py", line 39, in <module>
    from .storage import SparseStorage  # noqa
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/storage.py", line 7, in <module>
    from torch_sparse.utils import Final, index_sort
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/utils.py", line 5, in <module>
    import torch_sparse.typing
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch_sparse/typing.py", line 2, in <module>
    import pyg_lib  # noqa
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/pyg_lib/__init__.py", line 38, in <module>
    load_library('libpyg')
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/pyg_lib/__init__.py", line 35, in load_library
    torch.ops.load_library(spec.origin)
  File "/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/torch/_ops.py", line 573, in load_library
    ctypes.CDLL(path)
  File "/Users/semo/.pyenv/versions/3.10.9/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/libpyg.so, 0x0006): Library not loaded: /Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib
  Referenced from: <16F9F1F8-8305-3FAA-9EB1-C519E67F7DAC> /Users/semo/PycharmProjects/pretrained-pe/venv/lib/python3.10/site-packages/libpyg.so
  Reason: tried: '/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib' (no such file), '/Users/runner/hostedtoolcache/Python/3.10.8/x64/lib/libpython3.10.dylib' (no such file), '/usr/local/lib/libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)

Environment

pyg-lib version: 0.1.0/latest from source(0.1.0+pt113)
PyTorch version: 1.13.1/2.0.0
OS: macOS 13.2.1
Python version: 3.10.8
CUDA/cuDNN version: None/CPU (Intel Core i9)
How you installed PyTorch and pyg-lib (conda, pip, source): Both, see below
Any other relevant information:

Things I've tried:

Install torch & torch_geometric packages via conda and pyg_lib via pip in a clean env
Install torch, torch_geometric packages and pyg_lib via pip only
Test with PyTorch 2.0.0 and 1.13.1 (I'm positive the same setup worked with 1.13.1 and latest torch_geometric/pyg_lib last week)

semihcanturk commented 1 year ago

Turns out a (partial) solution is available in the pytorch_geometric repo, which is simply uninstalling pyg-lib: https://github.com/pyg-team/pytorch_geometric/issues/6530#issue-1559763529. Nevertheless keeping this issue open as this is likely to be undesirable behavior.

rusty1s commented 1 year ago

Please see my comment in https://github.com/pyg-team/pyg-lib/issues/211#issuecomment-1478934645.

semihcanturk commented 1 year ago

That didn't work unfortunately, but I managed to resolve the issues by reinstalling all packages from source. I think that way it updates the DYLD_LIBRARY_PATH correctly during setup.

pip install git+https://github.com/pyg-team/pyg-lib.git
pip install git+https://github.com/rusty1s/pytorch_sparse.git
pip install git+https://github.com/rusty1s/pytorch_scatter.git
pip install git+https://github.com/rusty1s/pytorch_cluster.git

rusty1s commented 5 months ago

I hit the same issue today :( I tried many different things, but in all scenarios the final binary was searching for a Python folder in /Library/Frameworks/Python.framework/Versions. Installing it from the universal install at https://www.python.org/downloads/macos/ fixed it for me.

Martins6 commented 1 week ago

It seems that even installing from Python official solves it, but the error message changes somewhat @rusty1s.

Traceback (most recent call last):
  File "/Users/adrielmartins/Documents/gcaptain/testingsomething.py", line 1, in <module>
    import pyg_lib
  File "/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/pyg_lib/__init__.py", line 39, in <module>
    load_library('libpyg')
  File "/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/pyg_lib/__init__.py", line 36, in load_library
    torch.ops.load_library(spec.origin)
  File "/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/torch/_ops.py", line 1295, in load_library
    ctypes.CDLL(path)
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: dlopen(/Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/libpyg.so, 0x0006): Symbol not found: __ZN3c1010Dispatcher17runRecordFunctionERN2at14RecordFunctionENSt3__117reference_wrapperIKNS_14FunctionSchemaEEENS_11DispatchKeyE
  Referenced from: <75FFC412-93B5-322B-8E6D-268DA3498CF4> /Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/libpyg.so
  Expected in:     <9C467D80-5BAC-3156-B480-F0FF7E45DEF2> /Users/adrielmartins/Documents/gcaptain/.venv/lib/python3.11/site-packages/torch/lib/libtorch_cpu.dylib

It seems that is still expecting packages elsewhere. I'm running this on M3 Pro Mac, using Python3.11.8, torch 2.3.0, torch_geometric >=2.5.3.

Do you have any other proposal, please?

pyg-team / pyg-lib

`import pyg_lib` fails due to `OSError: dlopen ... (no such file, not in dyld cache)` on macOS #217

🐛 Describe the bug

Environment