PyOpenCL tests seem to be broken on Ubuntu 23

[RABijl]

python version: 3.11.6 apt python3-numpy version: 1:1.24.2-1 python3-numpy includes numpy headers needed to prepare examples, running prepare examples in a virtualenv does help with this.

If prepare examples was not run in a virtualenv, the following errors show up:

platform linux -- Python 3.11.6, pytest-8.3.3, pluggy-1.5.0 -- /home/rabijl/Projects/pocl/build-test/examples/PyOpenCL/src/PyOpenCL-build/bin/python3                                         
cachedir: .pytest_cache                                                                                                                                                                       
rootdir: /home/rabijl/Projects/pocl/build-test/examples/PyOpenCL/src/PyOpenCL                                                                                                                 
configfile: pytest.ini                                                                                                                                                                        
collecting ...                                                                                                                                                                                
A module that was compiled using NumPy 1.x cannot be run in                                                                                                                                   
NumPy 2.1.1 as it may crash. To support both 1.x and 2.x                                                                                                                                      
versions of NumPy, modules must be compiled with NumPy 2.0.                                                                                                                                   
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.                                                                                                                           

If you are a user of the module, the easiest solution will be to                                                                                                                              
downgrade to 'numpy<2' or try to upgrade the affected module.                                                                                                                                 
We expect that some modules will need time to support NumPy 2.        

using a virtualenv with numpy version: 2.1.1 also doesn't work properly:

7/250 Test #289: PyOpenCL_test_clmath ..................................................................***Failed    0.81 sec
============================= test session starts ==============================
platform linux -- Python 3.11.6, pytest-8.3.3, pluggy-1.5.0 -- /home/rabijl/Projects/pocl-3/build_test/examples/PyOpenCL/src/PyOpenCL-build/bin/python3
cachedir: .pytest_cache
rootdir: /home/rabijl/Projects/pocl-3/build_test/examples/PyOpenCL/src/PyOpenCL
configfile: pytest.ini
collecting ... collected 0 items / 1 error

==================================== ERRORS ====================================
_____________________ ERROR collecting test/test_clmath.py _____________________
ImportError while importing test module '/home/rabijl/Projects/pocl-3/build_test/examples/PyOpenCL/src/PyOpenCL/test/test_clmath.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_clmath.py:28: in <module>
    import pyopencl.array as cl_array
E   ModuleNotFoundError: No module named 'pyopencl'
=========================== short test summary info ============================
ERROR test_clmath.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

There are also a number of deprecation warnings that might hint at the source of the problem.

[linehill]

using a virtualenv with numpy version: 2.1.1 also doesn't work properly: ... E ModuleNotFoundError: No module named 'pyopencl'

Did you install pyopencl into your virtualenv environment?

[RABijl]

I did not do that explicitly, I'm under the impression that is part of make prepare_examples, I'm not seeing it without a virtualenv

[linehill]

It seems that make prepare_examples installs pyopencl. Did you create and source the virtualenv after the make prepare_examples command? If I recall correctly, sourcing newly created virtualenv does not have (any?) python packages installed by default.

[RABijl]

I used a virtualenv that I'd used before for some onnx related things and already had numpy installed. I also did the following things:

1. source the virtualenv
2. delete everything in the build dir
3. run cmake with enable_testsuites=tier1
4. run make prepare_examples
5. run make
6. run make check_tier1

[franz]

I've have tried to debug this a few months ago, and AFAICT this is a bug in PyOpenCL. Due to some mistake in pip specifications, the installation pulls in both old and new (1.x and 2.x) numpy packages. I also tried following the official instructions to install from source, same result. If you manage to make a clean install of PyOpenCL with its testsuite we can try to fix the PyOpenCL install scripts in PoCL, otherwise i think this just needs to be reported as bug to PyOpenCL...

[pjaaskel]

@inducer any insights on this?

[inducer]

I guess the first stop would be to see whether there's any sign of pyopencl in the virtualenv (/home/rabijl/Projects/pocl-3/build_test/examples/PyOpenCL/src/PyOpenCL-build/ in this case?), specifically in lib/python3.11/site-packages or so. If not, then something is up with the build process. If yes, then something is up with the importing of it.

[RABijl]

Thank you for getting back to us. I took a look and there was a second instance of the virtualenv that was being created during make prepare_examples. This virtualenv has the pyopencl package in it, so there seems to be an issue importing it. I'm adding a log of the errors during the tests that might prove helpful. The PyOpenCL commit being built is: a1fe5f7. check_tier1 log build (prepare_examples) log