search

ComputeCanada / software-stack

Repository to host issues relative to the Compute Canada software stack
12 stars 0 forks source link

List of 2023 modules with eggs that needs to be re-installed #137

Closed ccoulombe closed 5 months ago

ccoulombe commented 8 months ago

DEPRECATION: Loading egg at X is deprecated. pip 23.3 will enforce this behaviour change. A possible replacement is to use pip for package installation..

PyPA says

The recommended action is to uninstall the distribution and reinstall it with a recent pip. This was planned for pip 23.3, but now is for 24.3. https://github.com/pypa/pip/issues/12330

Modules containings eggs format, which will be deprecated in pip 24.3. The solution is to rebuild them with pip install rather than python setup.py install.

When we upgrade pip, this will cause issues (error or blindly ignore the package).

Modules in 2023:

April 5th update:

Files with only .egg:

.../eccodes/2.31.0/lib/python3.11/site-packages/findlibs-0.0.5-py3.11.egg
.../eccodes/2.31.0/lib/python3.11/site-packages/eccodes-1.6.1-py3.11.egg
.../eccodes/2.31.0/lib/python3.11/site-packages/cfgrib-0.9.10.4-py3.11.egg
.../eccodes/2.31.0/lib/python3.11/site-packages/click-8.1.7-py3.11.egg
.../eccodes/2.31.0/lib/python3.10/site-packages/click-8.1.7-py3.10.egg
.../eccodes/2.31.0/lib/python3.10/site-packages/cfgrib-0.9.10.4-py3.10.egg
.../eccodes/2.31.0/lib/python3.10/site-packages/eccodes-1.6.1-py3.10.egg
.../eccodes/2.31.0/lib/python3.10/site-packages/findlibs-0.0.5-py3.10.egg
.../igraph/0.10.7/lib/python3.11/site-packages/texttable-1.7.0-py3.11.egg
.../igraph/0.10.7/lib/python3.11/site-packages/igraph-0.10.8-py3.11-linux-x86_64.egg
.../igraph/0.10.7/lib/python3.10/site-packages/texttable-1.7.0-py3.10.egg
.../igraph/0.10.7/lib/python3.10/site-packages/igraph-0.10.8-py3.10-linux-x86_64.egg
.../qgis/3.28.12/lib/python3.11/site-packages/lxml-4.9.3-py3.11-linux-x86_64.egg
.../qgis/3.28.12/lib/python3.11/site-packages/psycopg2-2.9.9-py3.11-linux-x86_64.egg
.../qgis/3.28.12/lib/python3.11/site-packages/OWSLib-0.29.3-py3.11.egg
.../qgis/3.28.12/lib/python3.11/site-packages/PyYAML-6.0.1-py3.11-linux-x86_64.egg
.../qgis/3.28.12/lib/python3.10/site-packages/psycopg2-2.9.9-py3.10-linux-x86_64.egg
.../qgis/3.28.12/lib/python3.10/site-packages/lxml-4.9.3-py3.10-linux-x86_64.egg
.../qgis/3.28.12/lib/python3.10/site-packages/OWSLib-0.29.3-py3.10.egg
.../qgis/3.28.12/lib/python3.10/site-packages/PyYAML-6.0.1-py3.10-linux-x86_64.egg
.../postgresql/16.0/lib/python3.11/site-packages/psycopg2-2.9.9-py3.11-linux-x86_64.egg
.../postgresql/16.0/lib/python3.10/site-packages/psycopg2-2.9.9-py3.10-linux-x86_64.egg
.../proj/9.2.0/lib/python3.11/site-packages/pyproj-3.6.1-py3.11-linux-x86_64.egg
.../proj/9.2.0/lib/python3.10/site-packages/pyproj-3.6.1-py3.10-linux-x86_64.egg
.../python/3.11.5/lib/python3.11/test/test_importlib/data/example-21.12-py3.6.egg
.../python/3.10.13/lib/python3.10/test/test_importlib/data/example-21.12-py3.6.egg
mboisson commented 8 months ago

Can you add context in this issue about why the egg format is a problem ?

Also, can we prevent such to be built through some hook ?

ccoulombe commented 8 months ago

Can you add context in this issue about why the egg format is a problem ?

done

Also, can we prevent such to be built through some hook ?

We could, but using pip will be default in EB v5.

ccoulombe commented 7 months ago

Update as of Apr 04 :

/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.11/site-packages/findlibs-0.0.5-py3.11.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.11/site-packages/eccodes-1.6.1-py3.11.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.11/site-packages/cfgrib-0.9.10.4-py3.11.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.11/site-packages/click-8.1.7-py3.11.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.10/site-packages/click-8.1.7-py3.10.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.10/site-packages/cfgrib-0.9.10.4-py3.10.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.10/site-packages/eccodes-1.6.1-py3.10.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/eccodes/2.31.0/lib/python3.10/site-packages/findlibs-0.0.5-py3.10.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/lxml-4.9.3-py3.11-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/psycopg2-2.9.9-py3.11-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/OWSLib-0.29.3-py3.11.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/PyYAML-6.0.1-py3.11-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/psycopg2-2.9.9-py3.10-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/lxml-4.9.3-py3.10-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/OWSLib-0.29.3-py3.10.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/PyYAML-6.0.1-py3.10-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcccore/mariadb/11.5.0/lib/python3.11/site-packages/protobuf-4.21.12-py3.11-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcccore/mariadb/11.5.0/lib/python3.11/site-packages/mysql_connector_python-8.2.0-py3.11-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcccore/mariadb/11.5.0/lib/python3.10/site-packages/protobuf-4.21.12-py3.10-linux-x86_64.egg
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Compiler/gcccore/mariadb/11.5.0/lib/python3.10/site-packages/mysql_connector_python-8.2.0-py3.10-linux-x86_64.egg
ccoulombe commented 7 months ago

Here's a list of modules with .egg-info. Some are trivial to fix, but other will require more work. Some only need dist-info instead (opencv, vtk, faiss).

CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/pdb4amber-22.0-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/ParmEd-4.0.0-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/AmberUtils-21.0-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/sander-22.0-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/pyMSMT-22.0-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/edgembar-0.2-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/packmol_memgen-2023.8.8-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/pytraj-2.0.6-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/ndfes-1.8-py3.11.egg-info
CUDA/gcc12/openmpi4/cuda12.2/ambertools/23.5/lib/python3.11/site-packages/MMPBSA.py-16.0-py3.11.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.11/site-packages/opencv_contrib_python-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.11/site-packages/opencv_contrib_python_headless-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.11/site-packages/opencv_python-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.11/site-packages/opencv_python_headless-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.10/site-packages/opencv_contrib_python-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.10/site-packages/opencv_contrib_python_headless-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.10/site-packages/opencv_python-4.8.1.egg-info
CUDA/gcc12/cuda12.2/opencv/4.8.1/lib/python3.10/site-packages/opencv_python_headless-4.8.1.egg-info
CUDA/gcc12/cuda12.2/faiss/1.7.4/lib/python3.11/site-packages/faiss-gpu-1.7.4.egg-info
CUDA/gcc12/cuda12.2/faiss/1.7.4/lib/python3.11/site-packages/faiss-cpu-1.7.4.egg-info
CUDA/gcc12/cuda12.2/faiss/1.7.4/lib/python3.11/site-packages/faiss-1.7.4.egg-info
CUDA/gcc12/cuda12.2/faiss/1.7.4/lib/python3.10/site-packages/faiss-gpu-1.7.4.egg-info
CUDA/gcc12/cuda12.2/faiss/1.7.4/lib/python3.10/site-packages/faiss-cpu-1.7.4.egg-info
CUDA/gcc12/cuda12.2/faiss/1.7.4/lib/python3.10/site-packages/faiss-1.7.4.egg-info
Compiler/intel2023/vtk/9.3.0/lib/python3.11/site-packages/vtk-9.3.0.egg-info
Compiler/intel2023/vtk/9.3.0/lib/python3.10/site-packages/vtk-9.3.0.egg-info
Compiler/gcc12/vtk/9.3.0/easybuild/vtk-version.egg-info
Compiler/gcc12/vtk/9.3.0/lib/python3.11/site-packages/vtk-9.3.0.egg-info
Compiler/gcc12/vtk/9.3.0/lib/python3.10/site-packages/vtk-9.3.0.egg-info
Compiler/gcc12/gdal/3.7.2/lib/python3.11/site-packages/GDAL-3.7.2-py3.11.egg-info
Compiler/gcc12/gdal/3.7.2/lib/python3.10/site-packages/GDAL-3.7.2-py3.10.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.11/site-packages/opencv_python-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.11/site-packages/opencv_python_headless-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.11/site-packages/opencv_contrib_python-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.11/site-packages/opencv_contrib_python_headless-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.10/site-packages/opencv_python-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.10/site-packages/opencv_python_headless-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.10/site-packages/opencv_contrib_python-4.9.0.egg-info
Compiler/gcc12/opencv/4.9.0/lib/python3.10/site-packages/opencv_contrib_python_headless-4.9.0.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.11/site-packages/opencv_contrib_python-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.11/site-packages/opencv_contrib_python_headless-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.11/site-packages/opencv_python-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.11/site-packages/opencv_python_headless-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.10/site-packages/opencv_contrib_python-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.10/site-packages/opencv_contrib_python_headless-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.10/site-packages/opencv_python-4.8.1.egg-info
Compiler/gcc12/opencv/4.8.1/lib/python3.10/site-packages/opencv_python_headless-4.8.1.egg-info
Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/lxml-4.9.3-py3.11-linux-x86_64.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/psycopg2-2.9.9-py3.11-linux-x86_64.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/OWSLib-0.29.3-py3.11.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.11/site-packages/PyYAML-6.0.1-py3.11-linux-x86_64.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/psycopg2-2.9.9-py3.10-linux-x86_64.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/lxml-4.9.3-py3.10-linux-x86_64.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/OWSLib-0.29.3-py3.10.egg
Compiler/gcc12/qgis/3.28.12/lib/python3.10/site-packages/PyYAML-6.0.1-py3.10-linux-x86_64.egg
Compiler/gcccore/thrift/0.19.0/lib/python3.11/site-packages/thrift-0.19.0-py3.11.egg-info
Compiler/gcccore/thrift/0.19.0/lib/python3.10/site-packages/thrift-0.19.0-py3.10.egg-info
Compiler/gcccore/btllib/1.6.2/lib/btllib/python/btllib-1.6.2.egg-info
Compiler/gcccore/python/3.11.5/lib/python3.11/test/test_importlib/data/example-21.12-py3.6.egg
Compiler/gcccore/python/3.10.13/lib/python3.10/test/test_importlib/data/example-21.12-py3.6.egg
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/Cython-0.29.30-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/charset_normalizer-2.0.10-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/requests-2.27.1-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/numpy-1.23.5-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/certifi-2021.10.8-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/idna-3.3-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/pycodestyle-2.8.0-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/toml-0.10.2-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/urllib3-1.26.8-py3.10.egg-info
Core/blender/4.0.2/4.0/python/lib/python3.10/site-packages/autopep8-1.6.0-py3.10.egg-info
MPI/gcc12/openmpi4/petsc/3.20.0/lib/python3.11/site-packages/petsc4py-3.20.0-py3.11.egg-info
MPI/gcc12/openmpi4/petsc/3.20.0/lib/python3.10/site-packages/petsc4py-3.20.0-py3.10.egg-info
MPI/gcc12/openmpi4/slepc/3.20.1/lib/python3.11/site-packages/slepc4py-3.20.1-py3.11.egg-info
MPI/gcc12/openmpi4/slepc/3.20.1/lib/python3.10/site-packages/slepc4py-3.20.1-py3.10.egg-info
MPI/gcc12/openmpi4/psi4/1.9/lib/python3.11/site-packages/OptKing-0.2.1-py3.11.egg-info
MPI/gcc12/openmpi4/psi4/1.9/lib/python3.10/site-packages/OptKing-0.2.1-py3.10.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/pdb4amber-22.0-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/ParmEd-4.0.0-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/AmberUtils-21.0-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/sander-22.0-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/pyMSMT-22.0-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/edgembar-0.2-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/packmol_memgen-2023.8.8-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/pytraj-2.0.6-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/ndfes-1.8-py3.11.egg-info
MPI/gcc12/openmpi4/ambertools/23.5/lib/python3.11/site-packages/MMPBSA.py-16.0-py3.11.egg-info
1biancheng commented 6 months ago

Why not add a command to detect egg-format packages in a specified environment and replace them with the latest format?

ccoulombe commented 6 months ago

@1biancheng This is exactly what we are doing :)

1biancheng commented 6 months ago

@1biancheng This is exactly what we are doing :) Cool.