search

inducer / pyopencl

OpenCL integration for Python, plus shiny features
http://mathema.tician.de/software/pyopencl
Other
1.05k stars 241 forks source link

BUG in 2021.2.11: _cl.cpython-39-darwin.so is x86 on arm64 #530

Closed psobolewskiPhD closed 2 years ago

psobolewskiPhD commented 2 years ago

Describe the bug

Import error on arm64 MacOS with version 2021.2.11:

Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:07) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyopencl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/piotrsobolewski/Dev/miniforge3/envs/test-napari/lib/python3.9/site-packages/pyopencl/__init__.py", line 28, in <module>
    import pyopencl.cltypes  # noqa: F401
  File "/Users/piotrsobolewski/Dev/miniforge3/envs/test-napari/lib/python3.9/site-packages/pyopencl/cltypes.py", line 22, in <module>
    from pyopencl.tools import get_or_register_dtype
  File "/Users/piotrsobolewski/Dev/miniforge3/envs/test-napari/lib/python3.9/site-packages/pyopencl/tools.py", line 36, in <module>
    from pyopencl._cl import bitlog2  # noqa: F401
ImportError: dlopen(/Users/piotrsobolewski/Dev/miniforge3/envs/test-napari/lib/python3.9/site-packages/pyopencl/_cl.cpython-39-darwin.so, 0x0002): tried: '/Users/piotrsobolewski/Dev/miniforge3/envs/test-napari/lib/python3.9/site-packages/pyopencl/_cl.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_cl.cpython-39-darwin.so' (no such file), '/usr/lib/_cl.cpython-39-darwin.so' (no such file)

(this did not occur with 2021.2.6)

To Reproduce Steps to reproduce the behavior:

  1. Setup an env using conda with python 3.9.7 (on arm64 M1 macOS 12)
  2. pip install pyopencl
  3. launch python REPL
  4. import pyopencl
  5. See error

Expected behavior pyopencl should import and run

Environment (please complete the following information):

Additional context Worked fine on arm64 with version 2021.2.6 Edit: just tested 2021.2.10, that I never used before and everything works fine, so the issue is related to a change between 2021.2.10 and 11.

inducer commented 2 years ago

Thanks for the report! I I think this should be filed against https://github.com/conda-forge/pyopencl-feedstock instead. Could you create an issue there, post the link here and close this?

I only just merged the 2021.2.11 PR: https://github.com/conda-forge/pyopencl-feedstock/pull/78, which looks mostly innocuous aside from a conda->mamba change.

psobolewskiPhD commented 2 years ago

Actually, I just checked and using conda install pyopencl JustWorks. The problem is only with a pip: I was actually installing a napari plugin with pyopencl as a dependency, so pyopencl was installed via pip. pip install pyopencl==2021.2.10 is fine but pip install pyopencl gives the error above.

Edit: so in fact using pip pyopencl is built:

Collecting pyopencl
  Downloading pyopencl-2021.2.11.tar.gz (452 kB)
     |████████████████████████████████| 452 kB 2.2 MB/s            
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl) (1.21.5)
Requirement already satisfied: pytools>=2021.2.7 in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl) (2021.2.9)
Requirement already satisfied: platformdirs>=2.2.0 in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl) (2.4.1)
Requirement already satisfied: appdirs>=1.4.0 in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pytools>=2021.2.7->pyopencl) (1.4.4)
Building wheels for collected packages: pyopencl
  Building wheel for pyopencl (setup.py) ... done
  Created wheel for pyopencl: filename=pyopencl-2021.2.11-cp39-cp39-macosx_11_0_arm64.whl size=563457 sha256=a2bf9e488b65b446fb01ee4b16a7511685b8dd7720bfdfd32ff201709e999688
  Stored in directory: /private/var/folders/sy/y3_x27j14cxg3_vhycs6cr_h0000gn/T/pip-ephem-wheel-cache-gu4wbv7y/wheels/5b/d3/e0/a6326c3ad5f5d8878ed14be7ba7a6b483b7608ec32f5ecad8a
Successfully built pyopencl
Installing collected packages: pyopencl
Successfully installed pyopencl-2021.2.11

But despite "success" it gives the import error. 2021.2.10 is fine:

Collecting pyopencl==2021.2.10
  Downloading pyopencl-2021.2.10.tar.gz (451 kB)
     |████████████████████████████████| 451 kB 2.1 MB/s            
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl==2021.2.10) (1.21.5)
Requirement already satisfied: pytools>=2021.2.7 in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl==2021.2.10) (2021.2.9)
Requirement already satisfied: appdirs>=1.4.0 in ./Dev/miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl==2021.2.10) (1.4.4)
Building wheels for collected packages: pyopencl
  Building wheel for pyopencl (setup.py) ... done
  Created wheel for pyopencl: filename=pyopencl-2021.2.10-cp39-cp39-macosx_11_0_arm64.whl size=562772 sha256=e50ac755394f4d8c580a8ddb7e7ed8bff2abb043ff3bc27980af660e79548768
  Stored in directory: /private/var/folders/sy/y3_x27j14cxg3_vhycs6cr_h0000gn/T/pip-ephem-wheel-cache-wj1qp8rg/wheels/12/ac/50/0790693e0360f14c20367832e3e843a18db02b6fa20c621a2c
Successfully built pyopencl
Installing collected packages: pyopencl
Successfully installed pyopencl-2021.2.10
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:07) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyopencl
>>>
psobolewskiPhD commented 2 years ago

If I clone the repo and do pip install . it imports no problem, so I guess it's something specific to what's on pypi. Edit: all the previous releases on pypi have linux wheels, but 2021.2.11 does not: https://pypi.org/project/pyopencl/2021.2.11/#files I can't say I know how this happens, but this pypi release seems suspect!

inducer commented 2 years ago

Edit: all the previous releases on pypi have linux wheels, but 2021.2.11 does not:

Turns out I forgot to push the tag, which is what triggers the wheel build. Should be going now, wheels should show up in a bit.

psobolewskiPhD commented 2 years ago

and now it's working fine again! Fresh conda 3.9.9 env. Thanks! Strange issue though, why would triggering the wheel build affect the build from source? Either way...

╰─ pip install --no-cache pyopencl                               (napari-env) ─╯
Collecting pyopencl
  Downloading pyopencl-2021.2.11.tar.gz (452 kB)
     |████████████████████████████████| 452 kB 2.0 MB/s            
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in ./miniforge3/envs/napari-env/lib/python3.9/site-packages (from pyopencl) (1.21.5)
Collecting pytools>=2021.2.7
  Downloading pytools-2021.2.9.tar.gz (66 kB)
     |████████████████████████████████| 66 kB 4.6 MB/s             
  Preparing metadata (setup.py) ... done
Collecting platformdirs>=2.2.0
  Downloading platformdirs-2.4.1-py3-none-any.whl (14 kB)
Requirement already satisfied: appdirs>=1.4.0 in ./miniforge3/envs/napari-env/lib/python3.9/site-packages (from pytools>=2021.2.7->pyopencl) (1.4.4)
Building wheels for collected packages: pyopencl, pytools
  Building wheel for pyopencl (setup.py) ... done
  Created wheel for pyopencl: filename=pyopencl-2021.2.11-cp39-cp39-macosx_11_0_arm64.whl size=563457 sha256=f862a985319fcb7777e9a69e7eecb7dacce5265aaf7a3b62685f16602c112728
  Stored in directory: /private/var/folders/sy/y3_x27j14cxg3_vhycs6cr_h0000gn/T/pip-ephem-wheel-cache-_c6yyzp6/wheels/5b/d3/e0/a6326c3ad5f5d8878ed14be7ba7a6b483b7608ec32f5ecad8a
  Building wheel for pytools (setup.py) ... done
  Created wheel for pytools: filename=pytools-2021.2.9-py2.py3-none-any.whl size=62370 sha256=b60393d06a575582a60d8d9ab072a6ad09576cde956b32e2241b6cad925ec8ac
  Stored in directory: /private/var/folders/sy/y3_x27j14cxg3_vhycs6cr_h0000gn/T/pip-ephem-wheel-cache-_c6yyzp6/wheels/e3/d5/07/ef3b818b787f6b3ef4b25d38075d7112aa7dcee48059887a49
Successfully built pyopencl pytools
Installing collected packages: pytools, platformdirs, pyopencl
Successfully installed platformdirs-2.4.1 pyopencl-2021.2.11 pytools-2021.2.9
╭─ ~/dev ·························································· ✔  50s ─╮
╰─ python                                                        (napari-env) ─╯
Python 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:07) 
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyopencl
>>>