matthew-brett / delocate

Find and copy needed dynamic libraries into python wheels
BSD 2-Clause "Simplified" License
262 stars 59 forks source link

AttributeError: partially initialized module 'json' has no attribute 'loads' (most likely due to a circular import) #171

Closed rdbisme closed 1 year ago

rdbisme commented 1 year ago

Describe the bug Relocating a wheel that links to pyarrow, causes partially initialized json module.

To Reproduce I have a wheel that links against pyarrow. On linux, auditwheel works as expected. When instead running relocate-wheel on macOS, I get an import error. The problems seems to occur with a weird collision against pyarrow/_json.pyx module.

Expected behavior Relocate wheel should be able to run without problems.

Wheels used This is the branch I'm building: https://github.com/0x0L/pgeon/pull/7

Platform (please complete the following information):

Additional context

/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/built_wheel/pgeon-0.1.dev1+g493e181-cp39-cp39-macosx_10_13_x86_64.whl x86_64
  delocate wrapper script
  Traceback (most recent call last):
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/bin/delocate-wheel", line 5, in <module>
      from delocate.cmd.delocate_wheel import main
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/delocate/__init__.py", line 3, in <module>
      from . import _version
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/delocate/_version.py", line 7, in <module>
      import json
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 106, in <module>
      from .decoder import JSONDecoder, JSONDecodeError
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 5, in <module>
      from json import scanner
    File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/scanner.py", line 5, in <module>
      from _json import make_scanner as c_make_scanner
    File "pyarrow/_json.pyx", line 1, in init pyarrow._json
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/pyarrow/__init__.py", line 65, in <module>
      import pyarrow.lib as _lib
    File "pyarrow/lib.pyx", line 24, in init pyarrow.lib
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/numpy/__init__.py", line 140, in <module>
      from . import core
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/numpy/core/__init__.py", line 9, in <module>
      from numpy.version import version as __version__
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/numpy/version.py", line 7, in <module>
      vinfo: dict[str, str] = get_versions()
    File "/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/cibw-run-4cufqzf0/cp39-macosx_x86_64/build/venv/lib/python3.9/site-packages/numpy/_version.py", line 21, in get_versions
      return json.loads(version_json)
  AttributeError: partially initialized module 'json' has no attribute 'loads' (most likely due to a circular import)

The wrapper script for the repair step in cibuildwheel is simply:

set -e

echo "delocate wrapper script"

export DYLD_LIBRARY_PATH=$(python -c 'import pyarrow as pa; print(":".join(pa.get_library_dirs()))')

args=("$@")
dest_dir=${args[0]}
wheel=${args[1]}
delocate_arch=${args[2]}

delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}
rdbisme commented 1 year ago

Not a bug. In order to make relocate work, I was exporting DYLD_LIBRARY_PATH. But then, it also finds the pyarrow _json compiled extension in the PATH, and it overrides the one from cpython.