Closed earnone closed 5 years ago
@maritsandstad have you tested extreme_events with a recent environment?
I believe the problem to be either transitory or if it persists, it's got to do with something else. In any case, can you plss run the following:
# cdo test, check that it supports hdf5
cdo --version
echo 0 | cdo -f nc input,r1x1 tmp.nc
ncdump tmp.nc | ncgen -k hdf5 -o tmp.nc
cdo -f nc copy tmp.nc tmp2.nc
and let me know what cdo
version you have and if all steps go through? :beer:
OK, send me that anyway but I am noticing the same problem from the command line:
cdo remapcon tmp.nc tmp2.nc
so it's not a fluke (damn!). Will investigate...
cdo remapcon t2.nc out.nc
cdo remapcon : Enter grid description file or name > tx.nc
cdo remapcon: YAC first order conservative weights from lonlat (144x96) to gaussian (192x96) grid
cdo remapcon: Processed 25671168 values from 1 variable over 1857 timesteps [6.91s 33MB]
cdo remapcon t1.nc out.nc
cdo remapcon : Enter grid description file or name > t1.nc
cdo remapcon: YAC first order conservative weights from gaussian (192x96) to gaussian (192x96) grid
cdo remapcon: Processed 34504704 values from 1 variable over 1872 timesteps [5.09s 23MB]
cdo remapcon t2.nc out.nc
cdo remapcon : Enter grid description file or name > t1.nc
cdo remapcon: YAC first order conservative weights from lonlat (144x96) to gaussian (192x96) grid
cdo remapcon: Processed 25671168 values from 1 variable over 1857 timesteps [8.34s 32MB]
cdo remapcon t2.nc out.nc
cdo remapcon : Enter grid description file or name > t2.nc
Warning! ***HDF5 library version mismatched error***
KABOOM!
so it seems it really doesn't like remapping from lonlat()
to lonlat()
Indeed all works fine in an environment with:
dependencies:
- hdf5=1.10.4
- cdo=1.9.6
and by forcing hdf5=1.10.5
in the environment cdo-1.9.7.1
will be installed and all works fine again; so it looks like the last version of hdf5 that cdo 1.9.6 was compiled against was 1.10.4; from there onwards cdo 1.9.7+ is compiled against hdf5=1.10.5
Again, a classic cockup from lovely conda :angry:
Confirm that cdo=1.9.7.1
+ hdf5=1.10.5
works well; I will pin cdo
in the environment file to >1.9.6
and recreate the environment
Bloody packages are misaligned again: requesting cdo>1.9.6
:
Package mpi conflicts for:
esmpy -> esmf[version='7.1.0.*,7.1.0r.*,7.1.0r'] -> mpich[version='>=3.2,<3.3.0a0,>=3.2.1,<3.3.0a0'] -> mpi==1.0=mpich
esmvalcore[version='>=2.0.0b0,<2.1'] -> esmpy -> esmf[version='7.1.0.*,7.1.0r.*,7.1.0r'] -> mpich[version='>=3.2,<3.3.0a0,>=3.2.1,<3.3.0a0'] -> mpi==1.0=mpich
ncl[version='>=6.5.0'] -> esmf -> mpich[version='>=3.2,<3.3.0a0,>=3.2.1,<3.3.0a0'] -> mpi==1.0=mpich
nco -> esmf -> mpich[version='>=3.2,<3.3.0a0,>=3.2.1,<3.3.0a0'] -> mpi==1.0=mpich
cdo[version='>1.9.6'] -> fftw=[build=mpi_openmpi_*] -> openmpi[version='>=3.1.3,<3.2.0a0'] -> mpi==1.0=openmpi
imagemagick -> fftw=[build=mpi_mpich_*] -> openmpi[version='>=3.1.3,<3.2.0a0'] -> mpi==1.0=openmpi
OK so we are pretty much hosed: hdf5
needs to be at 1.10.5
since a lot of the other deps have evolved and compiled against it, however cdo=1.9.7+
is incompatible with a lot of the other deps (see above); setting its version to 1.9.6
(or not setting the version at all, since conda will default it to 1.9.6
anyway) will allow for environment creation but the remap function will not work. Bloody Catch 22
workaround
export HDF5_DISABLE_VERSION_CHECK=1
will disable the internal cdo check for hdf5 versioning and cdo
will actually perform the remap operation correctly (albeit spitting the warning message); we need to contact the cdo
guys tell them to either completely disable the check from source (if indeed, as it seems, cdo=1.9.6 is compiled against both hdf5=1.10.4 and 1.10.5) and sort their new 1.9.7.1 version out
created an issue on the cdo's issue tracker forum: https://code.mpimet.mpg.de/boards/1/topics/7830
Summary of issue (changed original name to reflect the current state of the issue)
hdf5=1.10.5
is needed for a bunch of dependencies and it is not recommended to switch back to hdf5=1.10.4
;cdo=1.9.6
is compiled against hdf5=1.10.5
(eg cdo -V: Libraries: HDF5/1.10.4(1.10.5)
) but will throw a version mismatch warning if using cdo remapcon
; turning off the warnings by setting export HDF5_DISABLE_VERSION_CHECK=1
will allow the function to work, but it is not ideal by far);cdo=1.9.7.1
(that is fully compiled and built against hdf5=1.10.5
- I tested it proper) is not possible at the moment because conda conflict with mpi/mpich
, the new cdo needs mpi/openmpi
:
---
name: cdoenv
channels:
- conda-forge
dependencies:
throws:
Solving environment: failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Package mpi conflicts for: cdo=1.9.7.1 -> eccodes -> libnetcdf[version='>=4.6.2,<4.6.3.0a0,>=4.6.2,<4.7.0a0'] -> hdf5[version='>=1.10.4,<1.10.5.0a0,>=1.10.5,<1.10.6.0a0'] -> openmpi[version='>=3.1,<3.2.0a0'] -> mpi==1.0=openmpi hdf5=1.10.5 -> openmpi[version='>=3.1,<3.2.0a0'] -> mpi==1.0=openmpi mpich -> mpi==1.0=mpich
- no permanent solution found just yet
problem found: cdo=1.9.7.1 has a requirement on fftw which is hard set to openmpi in the package info/index.json. I asked for a relaxation of requirements at https://code.mpimet.mpg.de/boards/1/topics/7830?r=7848
closing this since it's got too much fluff in it, I opened a condesned issue summarizing the problem at #1239
recipe_quantilebias.yml
(default recipe in the development branch) is broken after installing the latest esmvaltool environment. There is an apparent conflict of hdf5 libraries when calling thecdo remapcon
command. This is the log: log_quantilebias_hdf5_error.txtThe only thing I can see that has changed since my previous environment is the hdf5 version: PREVIOUS:
LATEST:
Anyone having similar troubles? Should we then specify the hdf5 version in the environment.yml?