Closed sebhahn closed 4 years ago
jdkloe
commented
4 years ago Thanks for your report. So if this seems a tricky one to debug if it does not always occur or reproduce. Could you give me some details on the environment inside the VM that you use? Which exact pip3 and python3 version, which numpy and gcc/gfortran version do you use? Do you have any custom PYTHON_PATH or LD_LIBRARY_PATH settings that could be of influence?
sebhahn
commented
4 years ago CI settings
test_py36:
stage: test
script:
- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- conda config --set always_yes yes --set changeps1 no
- conda create -n warp_h_nrt_env python=3.6
- conda update -n base -c defaults conda
- conda env update -f environment.yml -n warp_h_nrt_env
- source activate warp_h_nrt_env
- pip install .
- python setup.py test
except:
- tagsenvironment.yml
name: warp_h_nrt_env
channels:
- default
- conda-forge
dependencies:
- numpy
- scipy
- netCDF4
- matplotlib
- cartopy
- pyresample
- gdal
- pip
- pip:
- h5py
- pybufr-ecmwf
- ascat
- pyraster
- Equi7Grid
- pytileprojThanks for providing the extra information. Unfortunately, debugging a thing like this is very hard if it does not reproduce, and on my side it does not. I replayed your conda configuration in a container (based on conda/miniconda3-centos7), and running it it passes the pybufr-ecmwf section without errors and builds its wheel as it should. After that the ascat wheel generation also runs fine, but on my side the script halts when it tries to download pyraster and produces this error:
Pip subprocess error:
ERROR: Could not find a version that satisfies the requirement pyraster (from -r /tmp/condaenv.fhumsta7.requirements.txt (line 4)) (from versions: none)
ERROR: No matching distribution found for pyraster (from -r /tmp/condaenv.fhumsta7.requirements.txt (line 4))If I remove the last 3 packages from the environment.yml script (pyraster, Equi7Grid and pytileproj) and repeat the procedure, then it runs successfully until the end.
So at this point I can only guess that there must be some interaction between these components, or with the specific conda version that you use. Your problem must be some alteration of the environment, since judging from your log the compilation ran without errors, and only when the script tried to load the ecmwfbufr module it stopped.
sebhahn
commented
4 years ago pyraster is an internal package, but the other two should be on pypi. In any case, I don't think the problem is related to these packages. I also have the feeling that the problem is somehow related to the conda environment, but interestingly other CI setups without pybufr show no problems.
Thanks for looking at it! I will make some more tests with different conda versions and setups.
In our internal continuous integration tests we experience occasionally the error
ModuleNotFoundError: No module named 'ecmwfbufr'. We install pybufr-ecmwf via pip on a linux VM. However, sometimes it runs through without any exception - with the same settings though.Full stack trace: