Changing default version of Python from 3.8 to 3.10

[mboisson]

Aiming to do it by early April 2023.

Things to do

• [x] Identify modules that have python bindings and do not support python 3.10
• [x] Update the modules below as possible
• [x] Add a warning message that the default version of python will change (done in PR https://github.com/ComputeCanada/software-stack-config/pull/45)

[mboisson]

436 modules have python bindings that do not include python/3.10:

$ grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep -v "python/3.10\|.bak_" | cut -d':' -f1 | wc
    436     436   40525

These correspond to 277 software (including multiple architecture builds):

$ for f in $(grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep -v "python/3.10\|.bak_" | cut -d':' -f1); do dirname $f; done | sort -u | wc
    277     277   22976

121 of those are for avx2:

$ for f in $(grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep -v "python/3.10\|.bak_" | cut -d':' -f1); do dirname $f; done | sort -u | grep "/avx2/" | wc
    121     121   10026

By software name, it is 63 pieces of software:

 for d in $(for f in $(grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep -v "python/3.10\|.bak_" | cut -d':' -f1); do dirname $f; done | sort -u); do basename $d; done | sort -u | tr '\n' ' '

arrayfire arrow autodiff blender boost boost-core boost-mpi bullet cantera eccodes esmf faiss gdal geos graph-tool gudhi hdf5 hdf5-mpi igraph ipykernel itk mariadb meep minimap2 mpi4py neuron nlopt ogre openbabel openbabel-omp opencv openmm openmm-alphafold openmm-plumed paraview paraview-offscreen paraview-offscreen-gpu petsc plplot plumed portaudio postgresql proj psi4 pyne pytest python-build-bundle qgis qscintilla qt quast rdkit scipy-stack slepc sumo symengine thrift trilinos triqs viennarna vtk vtk-mpi wxwidgets

[mboisson@build-node ~]$ for d in $(for f in $(grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep -v "python/3.10\|.bak_" | cut -d':' -f1); do dirname $f; done | sort -u); do basename $d; done | sort -u | wc
     63      63     524

Comparatively, we have 40 software packages with at least one version which supports python/3.10:

$ for d in $(for f in $(grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep "python/3.10" | cut -d':' -f1); do dirname $f; done | sort -u); do basename $d; done | sort -u | tr '\n' ' '
arrow boost clang cuquantum cython eccodes fmm3d gdal graph-tool hdf5 hdf5-mpi hoomd-blue hoomd-blue-mpi igraph ipykernel mpi4py mujoco netcdf netcdf-mpi oldest-supported-numpy opencv pairix parasail petsc petsc-64bits petsc-pardiso petsc-pardiso-64bits plumed postgresql proj pytest python-build-bundle reframe scipy-stack symengine thrift trilinos triqs vtk xtb [mboisson@build-node ~]$ for d in $(for f in $(grep -r "Compatible modules" /cvmfs/soft.computecanada.ca/easybuild/modules/2020 | grep python | grep "python/3.10" | cut -d':' -f1); do dirname $f; done | sort -u); do basename $d; done | sort -u | tr '\n' ' ' | wc
      0      40     364

[mboisson]

Excluding the intersection of these two lists, we have the following packages which do not have a single version which supports python 3.10:

• [x] arrayfire (Maxime, version 3.8.2)
• [x] autodiff (charles)
• [x] ~blender~ (not needed, wait for a request for a more recent version)
• [x] boost-core (Maxime, version 1.80.0)
• [x] boost-mpi (Maxime, version 1.80.0)
• [x] bullet (Ali - Installed the version 3.24 - Deployed and in production)
• [x] cantera (Maxime, version 2.6.0)
• [x] esmf (Maxime, version 8.4.0)
• [x] faiss (charles)
• [x] ~geos~ (not needed, use shapely wheel instead)
• [x] gudhi (Maxime, version 3.7.1)
• [x] itk (Maxime, version 5.2.1)
• [x] mariadb (Maxime, version 10.6.12)
• [x] meep (Charles)
• [x] minimap2 (Ali)
• [ ] neuron (Charles)
• [x] nlopt (Hélène)
• [x] ogre (Ali - Installing the version 13.1.1)
• [x] openbabel (Oliver/Ali)
• [x] openbabel-omp (Oliver/Ali)
• [x] openmm (charles)
• [x] ~openmm-alphafold~ (not needed, with openmm)
• [ ] openmm-plumed
• [x] paraview (Maxime, 5.11.0)
• [x] paraview-offscreen (Maxime, 5.11.0)
• [x] paraview-offscreen-gpu (Maxime, 5.11.0)
• [x] ~plplot~ (not needed, was never actually built to support python)
• [x] portaudio (Ali - Installed the version 190700_20210406 - Deployed and in production)
• [x] psi4 (Pawel)
• [x] pyne (Maxime)
• [x] qgis
• [x] qscintilla (Maxime, version 2.11.6)
• [x] qt (installed qt 5.15.8)
• [x] quast (exotic, wait until request)
• [x] rdkit (charles)
• [x] slepc (Maxime)
• [x] sumo (Ali - Installed the version 1.15.0 - Deployed and in production)
• [x] viennarna (Ali - Installed the version 2.5.1 - Deployed and in production)
• [x] ~vtk-mpi~ (not needed ?)
• [x] wxwidgets (Maxime)

[ofisette]

GEOS no longer packages its own Python bindings and PyGEOS is no longer updated. Shapely is the recommended Python GEOS interface now, and we have up-to-date wheels that include Python 3.10 support. See https://libgeos.org/usage/bindings/ and links therein.

[mboisson]

GEOS no longer packages its own Python bindings and PyGEOS is no longer updated. Shapely is the recommended Python GEOS interface now, and we have up-to-date wheels that include Python 3.10 support. See https://libgeos.org/usage/bindings/ and links therein.

ah, so no need for GEOS having python bindings then.

[ccoulombe]

Neuron will be, once patched. OpenMM and OpenMM-alphafold is in my todo for Alphafold, now that Alphafold can use any version of OpenMM, the OpenMM-Alphafold module will no longer be required

[ofisette]

Python 3.10 support was added to Qt in 5.15.5, which we currently do not have installed. After Qt and QScintilla are updated, I can update QGIS and add support for Python 3.10.

[mboisson]

@ofisette, Qt5 5.15.8 and QScintilla 2.11.6 now support Python 3.8, 3.9, 3.10.

[mboisson]

Neuron and Openmm-Plumed should not be blockers