Python 3.8 tests fail

[jaycedowell]

I've recently started seeing failures on the self-hosted Python 3.8 test cases. They seem to be from a problem building/installing pycuda. On master I see:

Building wheels for collected packages: pycuda
  Building wheel for pycuda (pyproject.toml): started
  Building wheel for pycuda (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error

  × Building wheel for pycuda (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [144 lines of output]
      *************************************************************
      *** I have detected that you have not run configure.py.
      *************************************************************
      *** Additionally, no global config files were found.
      *** I will go ahead with the default configuration.
      *** In all likelihood, this will not work out.
      ***
      *** See README_SETUP.txt for more information.
      ***
      *** If the build does fail, just re-run configure.py with the
      *** correct arguments, and then retry. Good luck!
      *************************************************************
      *** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT
      *************************************************************
      Continuing in 10 seconds...
      Continuing in 9 seconds...
Failed to build pycuda
      Continuing in 8 seconds...
      Continuing in 7 seconds...
      Continuing in 6 seconds...
      Continuing in 5 seconds...
      Continuing in 4 seconds...
      Continuing in 3 seconds...
      Continuing in 2 seconds...
      Continuing in 1 seconds...
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-38
      creating build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/curandom.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/_cluda.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/autoprimaryctx.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/_mymako.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/debug.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/__init__.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/gpuarray.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/compiler.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/cumath.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/driver.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/characterize.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/reduction.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/elementwise.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/tools.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/autoinit.py -> build/lib.linux-x86_64-cpython-38/pycuda
      copying pycuda/scan.py -> build/lib.linux-x86_64-cpython-38/pycuda
      creating build/lib.linux-x86_64-cpython-38/pycuda/gl
      copying pycuda/gl/__init__.py -> build/lib.linux-x86_64-cpython-38/pycuda/gl
      copying pycuda/gl/autoinit.py -> build/lib.linux-x86_64-cpython-38/pycuda/gl
      creating build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/pkt_build.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/cg.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/inner.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/coordinate.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/__init__.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/operator.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      copying pycuda/sparse/packeted.py -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      creating build/lib.linux-x86_64-cpython-38/pycuda/compyte
      copying pycuda/compyte/array.py -> build/lib.linux-x86_64-cpython-38/pycuda/compyte
      copying pycuda/compyte/__init__.py -> build/lib.linux-x86_64-cpython-38/pycuda/compyte
      copying pycuda/compyte/dtypes.py -> build/lib.linux-x86_64-cpython-38/pycuda/compyte
      running egg_info
      writing pycuda.egg-info/PKG-INFO
      writing dependency_links to pycuda.egg-info/dependency_links.txt
      writing requirements to pycuda.egg-info/requires.txt
      writing top-level names to pycuda.egg-info/top_level.txt
      reading manifest file 'pycuda.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      /tmp/pip-build-env-2ezrxqkg/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py:[264](https://github.com/ledatelescope/bifrost/actions/runs/3136219742/jobs/5092859199#step:7:265): UserWarning: Unknown distribution option: 'test_requires'
        warnings.warn(msg)
      warning: no files found matching 'doc/source/*.rst'
      warning: no files found matching 'doc/source/conf.py'
      warning: no files found matching 'doc/source/_static/*.css'
      warning: no files found matching 'doc/source/_templates/*.html'
      warning: no files found matching '*.cpp' under directory 'bpl-subset/bpl_subset/boost'
      warning: no files found matching '*.html' under directory 'bpl-subset/bpl_subset/boost'
      warning: no files found matching '*.inl' under directory 'bpl-subset/bpl_subset/boost'
      warning: no files found matching '*.txt' under directory 'bpl-subset/bpl_subset/boost'
      warning: no files found matching '*.h' under directory 'bpl-subset/bpl_subset/libs'
      warning: no files found matching '*.ipp' under directory 'bpl-subset/bpl_subset/libs'
      warning: no files found matching '*.pl' under directory 'bpl-subset/bpl_subset/libs'
      adding license file 'LICENSE'
      writing manifest file 'pycuda.egg-info/SOURCES.txt'
      creating build/lib.linux-x86_64-cpython-38/pycuda/cuda
      copying pycuda/cuda/pycuda-complex-impl.hpp -> build/lib.linux-x86_64-cpython-38/pycuda/cuda
      copying pycuda/cuda/pycuda-complex.hpp -> build/lib.linux-x86_64-cpython-38/pycuda/cuda
      copying pycuda/cuda/pycuda-helpers.hpp -> build/lib.linux-x86_64-cpython-38/pycuda/cuda
      copying pycuda/sparse/pkt_build_cython.pyx -> build/lib.linux-x86_64-cpython-38/pycuda/sparse
      running build_ext
      building '_driver' extension
      creating build/temp.linux-x86_64-cpython-38
      creating build/temp.linux-x86_64-cpython-38/bpl-subset
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/python
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/python/src
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/python/src/converter
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/python/src/object
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/smart_ptr
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/smart_ptr/src
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/system
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/system/src
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/thread
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/thread/src
      creating build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/thread/src/pthread
      creating build/temp.linux-x86_64-cpython-38/src
      creating build/temp.linux-x86_64-cpython-38/src/cpp
      creating build/temp.linux-x86_64-cpython-38/src/wrapper
      gcc -pthread -Wno-unused-result -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset -I/usr/local/cuda-11.2/include -I/tmp/pip-build-env-2ezrxqkg/overlay/lib/python3.8/site-packages/numpy/core/include -I/opt/hostedtoolcache/Python/3.8.13/x64/include/python3.8 -c bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.cpp -o build/temp.linux-x86_64-cpython-38/bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.o
      In file included from bpl-subset/bpl_subset/boost/python/detail/prefix.hpp:13:0,
                       from bpl-subset/bpl_subset/boost/python/handle.hpp:8,
                       from bpl-subset/bpl_subset/boost/python/converter/arg_to_python_base.hpp:7,
                       from bpl-subset/bpl_subset/libs/python/src/converter/arg_to_python_base.cpp:6:
      bpl-subset/bpl_subset/boost/python/detail/wrap_python.hpp:50:11: fatal error: pyconfig.h: No such file or directory
       # include <pyconfig.h>
                 ^~~~~~~~~~~~
      compilation terminated.
      /tmp/pip-build-env-2ezrxqkg/overlay/lib/python3.8/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'pycuda.cuda' as data is deprecated, please list it in `packages`.
          !!

          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'pycuda.cuda' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.

          'pycuda.cuda' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).

          Please make sure that 'pycuda.cuda' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" and "data files" on setuptools
          documentation page.

      !!

        check.warn(importable)
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pycuda
ERROR: Could not build wheels for pycuda, which is required to install pyproject.toml-based projects

I've tried rolling back the version of Python but that doesn't seem to help. Maybe I should see if there was a new pycuda release.

[jaycedowell]

Nope, pycuda is still at 2022.1.

[jaycedowell]

I think is some kind of problem related to the bundled version of boost that comes with pycuda. I tried installing libboost-all-dev from the Ubuntu repos but that didn't do anything. I'm trying a new version of setup-python to see if it was a change there.

[jaycedowell]

Updating actions/checkout and actions/skip-duplicate-actions at least gets rid of all of the deprecation warnings.

[jaycedowell]

Yeah, it's something about pycuda. When I don't install it things work, even on Py3.10.

[jaycedowell]

I'm out of ideas for today. @league any thoughts?

[jaycedowell]

Updating the actions runner to v2.299.1 doesn't seem to have helped.

[jaycedowell]

@league seems to have cracked this on the Py3.10 branch.