denisri
commented
3 years ago Oh. Very strange ! And very annoying...
Closed sapetnioc closed 3 years ago
denisri
commented
3 years ago Oh. Very strange ! And very annoying...
denisri
commented
3 years ago I'll try to build it again, we will see...
denisri
commented
3 years ago I can't rebuild the run image, there is an error when reinstalling h5py:
[...]
Successfully installed fastcluster-1.1.28
+ sudo CPPFLAGS=-I/usr/include/mpi python2 -m pip --no-cache-dir install --no-binary=h5py h5py<2.10
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting h5py<2.10
Downloading h5py-2.9.0.tar.gz (287 kB)
|################################| 287 kB 27.8 MB/s
Requirement already satisfied: numpy>=1.7 in /usr/local/lib/python2.7/dist-packages (from h5py<2.10) (1.16.6)
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages (from h5py<2.10) (1.16.0)
Skipping wheel build for h5py, due to binaries being disabled for it.
Installing collected packages: h5py
Attempting uninstall: h5py
Found existing installation: h5py 2.10.0
Uninstalling h5py-2.10.0:
Successfully uninstalled h5py-2.10.0
Running setup.py install for h5py ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7ZJR8d/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7ZJR8d/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-RxzubS/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python2.7/h5py
cwd: /tmp/pip-install-7ZJR8d/h5py/
Complete output (111 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/h5py
copying h5py/version.py -> build/lib.linux-x86_64-2.7/h5py
copying h5py/ipy_completer.py -> build/lib.linux-x86_64-2.7/h5py
copying h5py/highlevel.py -> build/lib.linux-x86_64-2.7/h5py
copying h5py/h5py_warnings.py -> build/lib.linux-x86_64-2.7/h5py
copying h5py/__init__.py -> build/lib.linux-x86_64-2.7/h5py
creating build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/vds.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/selections2.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/selections.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/group.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/filters.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/files.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/dims.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/datatype.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/dataset.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/compat.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/base.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/attrs.py -> build/lib.linux-x86_64-2.7/h5py/_hl
copying h5py/_hl/__init__.py -> build/lib.linux-x86_64-2.7/h5py/_hl
creating build/lib.linux-x86_64-2.7/h5py/tests
copying h5py/tests/common.py -> build/lib.linux-x86_64-2.7/h5py/tests
copying h5py/tests/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests
creating build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_slicing.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_selections.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_objects.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_h5t.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_h5p.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_h5f.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_h5.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_group.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_file_image.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_file.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_datatype.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_dataset.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_base.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/test_attrs.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
copying h5py/tests/old/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests/old
creating build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_threads.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_filters.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_file.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_datatype.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
copying h5py/tests/hl/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl
creating build/lib.linux-x86_64-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl/test_vds
copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-x86_64-2.7/h5py/tests/hl/test_vds
running build_ext
Autodetection skipped [invalid syntax (pkgconfig.py, line 281)]
********************************************************************************
Summary of the h5py configuration
Path to HDF5: None
HDF5 Version: '1.8.4'
MPI Enabled: False
Rebuild Required: True
********************************************************************************
Executing api_gen rebuild of defs
Executing cythonize()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-7ZJR8d/h5py/setup.py", line 168, in <module>
cmdclass = CMDCLASS,
File "/usr/lib/python2.7/dist-packages/setuptools/__init__.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 61, in run
return orig.install.run(self)
File "/usr/lib/python2.7/distutils/command/install.py", line 601, in run
self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "setup_build.py", line 202, in run
self.extensions = cythonize(self._make_extensions(config),
File "setup_build.py", line 80, in _make_extensions
import pkgconfig
File "build/bdist.linux-x86_64/egg/pkgconfig/__init__.py", line 1, in <module>
File "/tmp/pip-install-7ZJR8d/h5py/.eggs/pkgconfig-1.5.4-py2.7.egg/pkgconfig/pkgconfig.py", line 281
flags = _query(packages, *os_opts, *_build_options(option, static=static))
^
SyntaxError: invalid syntax
----------------------------------------
Rolling back uninstall of h5py
Moving to /usr/local/lib/python2.7/dist-packages/h5py-2.10.0.dist-info/
from /usr/local/lib/python2.7/dist-packages/~5py-2.10.0.dist-info
Moving to /usr/local/lib/python2.7/dist-packages/h5py/
from /usr/local/lib/python2.7/dist-packages/~5py
ERROR: Command errored out with exit status 1: /usr/bin/python2 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-7ZJR8d/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-7ZJR8d/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-RxzubS/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python2.7/h5py Check the logs for full command output.
FATAL: While performing build: while running engine: exit status 1It seems that h5py 2.10 was installed, then removed to install 2.9 as a requirement for something else (but I don't know what exactly), and h5py 2.9 cannot be installed anymore. Joys of pip, again...
denisri
commented
3 years ago and the actual error is in a local install of pkgconfig, which is not compatible with python2 any longer. I don't know how to fix that...
denisri
commented
3 years ago The install of h5py<2.10 is explicitly asked in the file install_pip_dependencies.sh, I don't know if 2.10 would work (and the problem is in pkgconfig apparently thus may shiw up at any other time)
sapetnioc
commented
3 years ago To make debug easier, I fixed the casa_distro_admin local_install so that it works with singularity. This enables step by step install in a writable image. Allowing to manually relaunch the command that cause problem. Here is how I start it for Singularity :
cd ~/casa_distro
singularity build --fakeroot --sandbox test_image casa-system-5.0.sif
mkdir test_image/casa_distro
singularity run --no-home --fakeroot --writable -B ~/casa_distro/brainvisa/src/development/casa-distro/master:/casa_distro test_image bash
# In container
apt update
apt install python sudo
adduser brainvisa
/casa_distro/bin/casa_distro_admin local_install type=run action=next
/casa_distro/bin/casa_distro_admin local_install type=run action=next
...Installation of pkgconfig from pip allowed me to install h5py<2.10. I will try to add this in the installation script.
ylep
commented
3 years ago This pip mess is a nightmare... We should try really hard to stick to APT-installed packages for the next Ubuntu 20 image. I will try to set up the Ubuntu 20 image in July :upside_down_face:
denisri
commented
3 years ago I could build a run image with h5py==2.10, but I don't know / remember why we specified h5py<2.10, if it was for a compatibility issue or just to have a hand on installed versions. Moreover:
Should I push the image on the server so that we can test it ? But it may also break some things for developers...
denisri
commented
3 years ago This
pipmess is a nightmare... We should try really hard to stick to APT-installed packages for the next Ubuntu 20 image. I will try to set up the Ubuntu 20 image in July upside_down_face
We may experience fewer problems by moving to python3 also: a number of our problems are caused by many modules dropping support for python2.
denisri
commented
3 years ago Installation of pkgconfig from pip allowed me to install h5py<2.10. I will try to add this in the installation script.
Sorry I had not seen that (I ran the build this morning then was in meetings all morning). If you could make pkgconfig work, then it's probably better to stick with h5py 2.9 as it was tested this way.
denisri
commented
3 years ago OK I could rebuild the run image using @sapetnioc's fix. I'll push it on the server.
Whereas it is in the installation file : https://github.com/brainvisa/casa-distro/blob/b9430a71c5c11c3323131588bfde0cb2c62e2185/share/docker/casa-run/ubuntu-18.04/install_apt_dependencies.sh#L322