search

ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.
https://www.esmvaltool.org
Apache License 2.0
42 stars 38 forks source link

Anomaly calculation for OBS got broken early march. #710

Closed bascrezee closed 1 year ago

bascrezee commented 4 years ago

Describe the bug Unfortunately, it seems like none of the tests has flagged (something to look into later I would say!). But for several observational datasets the calculation of anomalies goes wrong with non-physical values coming out of the preprocessor. I could track down the problems to 3-4 March 2020. With everything working fine on the 3rd of March (git checkout 'master@{2020-03-03}') and wrong results from 4 March onwards (git checkout 'master@{2020-03-04}'). To create the plots and run the recipe, one needs a specific ESMValTool branch: git checkout C3S_511_MPQB. But reproducing it and simply inspecting NetCDF output files would work as well of course. Since the changes in the anomalies preprocessor were authored by @jvegasbsc my hope is that he can solve this issue. It would be a good additional check if someone can reproduce the error (@hb326 @BenMGeo or @mattiarighi). The error also occurred for ERA5, a dataset that is more widely used. Tagging @hirschim just to keep you updated.

Fine: lineplot_sm_2015-2018_GOOD Generated using Copernicus Climate Change Service information 2020

Wrong: lineplot_sm_2015-2018_WRONG Generated using Copernicus Climate Change Service information 2020

Excerpt from the differences of ESMValCore between 3rd and 4th of March 2020: 

diff --git a/esmvalcore/preprocessor/_time.py b/esmvalcore/preprocessor/_time.py
index 1e29395e3..c1cf2071b 100644
--- a/esmvalcore/preprocessor/_time.py
+++ b/esmvalcore/preprocessor/_time.py
@@ -462,16 +462,15 @@ def anomalies(cube, period, reference):
     cube_time = cube.coord('time')
     ref = {}
     for ref_slice in reference.slices_over(ref_coord):
-        ref[ref_slice.coord(ref_coord).points[0]] = da.ravel(
-            ref_slice.core_data())
+        ref[ref_slice.coord(ref_coord).points[0]] = ref_slice.core_data()
+
     cube_coord_dim = cube.coord_dims(cube_coord)[0]
+    slicer = [slice(None)] * len(data.shape)
+    new_data = []
     for i in range(cube_time.shape[0]):
-        time = cube_time.points[i]
-        indexes = cube_time.points == time
-        indexes = iris.util.broadcast_to_shape(indexes, data.shape,
-                                               (cube_coord_dim, ))
-        data[indexes] = data[indexes] - ref[cube_coord.points[i]]
-
+        slicer[cube_coord_dim] = i
+        new_data.append(data[tuple(slicer)] - ref[cube_coord.points[i]])
+    data = da.stack(new_data, axis=cube_coord_dim)
     cube = cube.copy(data)
     cube.remove_coord(cube_coord)
     return cube

commit f61cc0e946dd4cd1de02fb738046f5273db16025
Author: Javier Vegas <javier.vegas@bsc.es>
Date:   Tue Jan 21 16:31:42 2020 +0100

    Remove print and extra coordinate

diff --git a/esmvalcore/preprocessor/_time.py b/esmvalcore/preprocessor/_time.py
index 612bfa47a..1e29395e3 100644
--- a/esmvalcore/preprocessor/_time.py
+++ b/esmvalcore/preprocessor/_time.py
@@ -411,7 +411,6 @@ def climate_statistics(cube, operator='mean', period='full'):
     operator = get_iris_analysis_operation(operator)
     clim_cube = cube.aggregated_by(clim_coord, operator)
     clim_cube.remove_coord('time')
-    print(clim_cube)
     if clim_cube.coord(clim_coord.name()).is_monotonic():
         iris.util.promote_aux_coord_to_dim_coord(clim_cube, clim_coord.name())
     else:
@@ -474,6 +473,7 @@ def anomalies(cube, period, reference):
         data[indexes] = data[indexes] - ref[cube_coord.points[i]]

     cube = cube.copy(data)
+    cube.remove_coord(cube_coord)
     return cube

Recipe: 

# ESMValTool
# recipe_anom_bug.yml
---
documentation:
  description: |
    Recipe for demonstrating a bug. To get wrong results run `git checkout ' master@{2020-03-04}'` in ESMValCore dir. To get good results run `git checkout 'master@{2020-03-03}'` in ESMValCore dir. Use branch `C3S_511_MPQB` from ESMValTool.

  authors:
    - crezee_bas

################################################
# Define some default parameters using anchors #
################################################

commongrid: &commongrid
  regrid:
    target_grid: 0.25x0.25
    scheme: nearest
  regrid_time:  # this is needed for a fully homogeneous time coordinate
    frequency: mon

icefreeland: &icefreeland
  mask_landsea:
    mask_out: sea
  mask_glaciated:
    mask_out: glaciated

commonmask: &commonmask  # should be preceded by commongrid
  mask_fillvalues:
    threshold_fraction: 0.0     # keep all missing values
    min_value: -1e20            # small enough not to alter the data

nonnegative: &nonnegative
  clip:
    minimum: 0.0

################################################
################################################
################################################

datasets_from_1992_2019: &datasets_from_1992_2019
  additional_datasets:
    - {dataset: CDS-SATELLITE-SOIL-MOISTURE, type: sat, project: OBS, mip: Lmon,
       version: CUSTOM-TCDR-ICDR-20200602, tier: 3, start_year: 2015, end_year: 2018}

#     - {dataset: CDS-SATELLITE-SOIL-MOISTURE, project: OBS, tier: 3, type: sat,
#        version: CUSTOM-TCDR-ICDR-20200602, start_year: 1992, end_year: 2019, mip: Lmon}
#     - {dataset: cds-era5-land-monthly, type: reanaly, project: OBS, mip: Lmon,
#        version: 1, tier: 3, start_year: 1992, end_year: 2019}
#     - {dataset: cds-era5-monthly, type: reanaly, project: OBS, mip: Lmon,
#        version: 1, tier: 3, start_year: 1992, end_year: 2019}
#     - {dataset: MERRA2, type: reanaly, project: OBS6, mip: Lmon,
#        version: 5.12.4, tier: 3, start_year: 1992, end_year: 2019}

preprocessors:
  pp_lineplots_ano:
    custom_order: true
    <<: *icefreeland
    <<: *commongrid
    <<: *commonmask
    <<: *nonnegative
    anomalies:
      period: monthly
      reference: [2015,2018]
      standardize: false
    area_statistics:
      operator: mean

diagnostics:
  lineplots_ano:
    variables:
      sm:
        preprocessor: pp_lineplots_ano
        mip: Lmon
    scripts:
      lineplot:
        script: mpqb/mpqb_lineplot.py
    <<: *datasets_from_1992_2019
jvegreg commented 4 years ago

Thanks for spotting this. It looks like there is a problem with the management of the NaNs, as the results are pretty close to the fill value. Let me investigate a bit

jvegreg commented 4 years ago

One question, @bascrezee

The input data is not available in Jasmin's common data pool?

bascrezee commented 4 years ago

I would expect it to be there, although maybe not precisely this version of the dataset. Who is in charge of the data at Jasmin?

jvegreg commented 4 years ago

@valeriupredoi. I only see ERA-Interim data in the Tier3.

mattiarighi commented 4 years ago

Which dataset(s) do you need?

jvegreg commented 4 years ago

This one:

 - {dataset: CDS-SATELLITE-SOIL-MOISTURE, type: sat, project: OBS, mip: Lmon,
       version: CUSTOM-TCDR-ICDR-20200602, tier: 3, start_year: 2015, end_year: 2018}
mattiarighi commented 4 years ago

Here https://swiftbrowser.dkrz.de/tcl_s/NRlFdyevrAzCG5

mattiarighi commented 4 years ago

Let me know if it works.

jvegreg commented 4 years ago

It works, a bit too well:

lineplot_sm_2015-2018 Generated using Copernicus Climate Change Service information 2020

bascrezee commented 4 years ago

What do you mean with 'too well'? Are you not able to reproduce the problem?

jvegreg commented 4 years ago

What do you mean with 'too well'? Are you not able to reproduce the problem?

Yes, the above plot is what I got when running the recipe with v2.0.0 branch for ESMValCore. The conda environment I used:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
_r-mutex                  1.0.1               anacondar_1    conda-forge
alabaster                 0.7.12                   pypi_0    pypi
antlr-python-runtime      4.7.2                 py38_1001    conda-forge
astroid                   2.3.3                    pypi_0    pypi
attrs                     19.3.0                     py_0    conda-forge
autodocsumm               0.1.13                   pypi_0    pypi
autopep8                  1.5.2                    pypi_0    pypi
babel                     2.8.0                    pypi_0    pypi
binutils_impl_linux-64    2.34                 h53a641e_0    conda-forge
binutils_linux-64         2.34                hc952b39_18    conda-forge
bokeh                     2.0.1            py38h32f6830_0    conda-forge
boost-cpp                 1.72.0               h8e57a91_0    conda-forge
brotlipy                  0.7.0           py38h1e0a361_1000    conda-forge
bwidget                   1.9.14                        0    conda-forge
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.4.5.1           hecc5488_0    conda-forge
cairo                     1.16.0            hcf35c78_1003    conda-forge
cartopy                   0.17.0          py38h9cf8511_1015    conda-forge
cdo                       1.5.3                    pypi_0    pypi
cdsapi                    0.2.7                    pypi_0    pypi
certifi                   2020.4.5.1       py38h32f6830_0    conda-forge
cf-units                  2.1.4            py38h8790de6_0    conda-forge
cffi                      1.14.0           py38hd463f26_0    conda-forge
cfitsio                   3.470                h3eac812_5    conda-forge
cftime                    1.1.2            py38h8790de6_0    conda-forge
chardet                   3.0.4           py38h32f6830_1006    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
click-plugins             1.1.1                      py_0    conda-forge
cligj                     0.5.0                    pypi_0    pypi
cloudpickle               1.4.1                      py_0    conda-forge
cmocean                   2.0                      pypi_0    pypi
codespell                 1.16.0                   pypi_0    pypi
colorama                  0.4.3                    pypi_0    pypi
coverage                  5.1                      pypi_0    pypi
cryptography              2.9.2            py38h766eaa4_0    conda-forge
curl                      7.69.1               h33f0ec9_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
cython                    0.29.17          py38h950e882_0    conda-forge
cytoolz                   0.10.1           py38h516909a_0    conda-forge
dask                      2.15.0                     py_0    conda-forge
dask-core                 2.15.0                     py_0    conda-forge
decorator                 4.4.2                      py_0    conda-forge
distributed               2.15.2           py38h32f6830_0    conda-forge
docutils                  0.16                     pypi_0    pypi
dodgy                     0.2.1                    pypi_0    pypi
eccodes                   2.17.0               h59f7be3_1    conda-forge
ecmwf-api-client          1.5.4                    pypi_0    pypi
entrypoints               0.3                      pypi_0    pypi
eofs                      1.4.0                    pypi_0    pypi
esmf                      8.0.0           mpi_mpich_h9a42a66_106    conda-forge
esmpy                     8.0.0           mpi_mpich_py38ha9b28fa_101    conda-forge
esmvalcore                2.0.0                     dev_0    <develop>
esmvaltool                2.0.0b4                   dev_0    <develop>
expat                     2.2.9                he1b5a44_2    conda-forge
fftw                      3.3.8           nompi_h7f3a6c3_1110    conda-forge
fiona                     1.8.13.post1             pypi_0    pypi
fire                      0.3.1                    pypi_0    pypi
flake8                    3.7.9                    pypi_0    pypi
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      2.001                hab24e00_0    conda-forge
font-ttf-source-code-pro  2.030                hab24e00_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.13.1            h86ecdb6_1001    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.10.1               he06d7ca_0    conda-forge
freexl                    1.0.5             h14c3975_1002    conda-forge
fribidi                   1.0.9                h516909a_0    conda-forge
fsspec                    0.7.3                      py_0    conda-forge
gcc_impl_linux-64         7.3.0                hd420e75_5    conda-forge
gcc_linux-64              7.3.0               h553295d_18    conda-forge
gdal                      3.0.4            py38h172510d_6    conda-forge
gdk-pixbuf                2.38.2               h3f25603_3    conda-forge
geos                      3.8.1                he1b5a44_0    conda-forge
geotiff                   1.5.1               h05acad5_10    conda-forge
gettext                   0.19.8.1          hc5be6a0_1002    conda-forge
gfortran_impl_linux-64    7.3.0                hdf63c60_5    conda-forge
gfortran_linux-64         7.3.0               h553295d_18    conda-forge
ghostscript               9.22              hf484d3e_1001    conda-forge
giflib                    5.2.1                h516909a_2    conda-forge
glib                      2.64.2               h6f030ca_0    conda-forge
gobject-introspection     1.58.2          py38h03d966d_1004    conda-forge
graphite2                 1.3.13            he1b5a44_1001    conda-forge
graphviz                  2.42.3               h0511662_0    conda-forge
gsl                       2.6                  h294904e_0    conda-forge
gxx_impl_linux-64         7.3.0                hdf63c60_5    conda-forge
gxx_linux-64              7.3.0               h553295d_18    conda-forge
harfbuzz                  2.4.0                h9f30f68_3    conda-forge
hdf4                      4.2.13            hf30be14_1003    conda-forge
hdf5                      1.10.5          mpi_mpich_ha7d0aea_1004    conda-forge
hdfeos2                   2.20              h64bfcee_1000    conda-forge
hdfeos5                   5.1.16               h8b6279f_5    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
html5lib                  1.0.1                      py_0    conda-forge
icu                       64.2                 he1b5a44_1    conda-forge
idna                      2.9                        py_1    conda-forge
imagemagick               7.0.10_11       pl526h201ca68_0    conda-forge
imagesize                 1.2.0                    pypi_0    pypi
iris                      2.4.0                    py38_0    conda-forge
isodate                   0.6.0                    pypi_0    pypi
isort                     4.3.21                   pypi_0    pypi
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jbig                      2.1               h516909a_2002    conda-forge
jinja2                    2.11.2             pyh9f0ad1d_0    conda-forge
joblib                    0.14.1                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
json-c                    0.13.1            h14c3975_1001    conda-forge
kealib                    1.4.13               hec59c27_0    conda-forge
keepalive                 0.5                        py_1    conda-forge
kiwisolver                1.2.0            py38hbf85e49_0    conda-forge
krb5                      1.17.1               h2fd8d38_0    conda-forge
lazy-object-proxy         1.4.3                    pypi_0    pypi
ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
libaec                    1.0.4                he1b5a44_1    conda-forge
libblas                   3.8.0               16_openblas    conda-forge
libcblas                  3.8.0               16_openblas    conda-forge
libcroco                  0.6.13               h8d621e5_1    conda-forge
libcurl                   7.69.1               hf7181ac_0    conda-forge
libdap4                   3.20.6               h1d1bd15_0    conda-forge
libedit                   3.1.20170329      hf8c457e_1001    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgdal                   3.0.4                h3dfc09a_6    conda-forge
libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
libgomp                   9.2.0                h24d8f2e_2    conda-forge
libiconv                  1.15              h516909a_1006    conda-forge
libkml                    1.3.0             hb574062_1011    conda-forge
liblapack                 3.8.0               16_openblas    conda-forge
libllvm8                  8.0.1                hc9558a2_0    conda-forge
libnetcdf                 4.7.4           mpi_mpich_h755db7c_1    conda-forge
libopenblas               0.3.9                h5ec1e0e_0    conda-forge
libpng                    1.6.37               hed695b0_1    conda-forge
libpq                     12.2                 h5513abc_1    conda-forge
librsvg                   2.48.4               h33a7fed_0    conda-forge
libspatialite             4.3.0a            h2482549_1038    conda-forge
libssh2                   1.8.2                h22169c7_2    conda-forge
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
libtiff                   4.1.0                hc7e4089_6    conda-forge
libtool                   2.4.6             h14c3975_1002    conda-forge
libunwind                 1.3.1             hf484d3e_1000    conda-forge
libuuid                   2.32.1            h14c3975_1000    conda-forge
libwebp                   1.1.0                h56121f0_4    conda-forge
libwebp-base              1.1.0                h516909a_3    conda-forge
libxcb                    1.13              h14c3975_1002    conda-forge
libxml2                   2.9.10               hee79883_0    conda-forge
libxslt                   1.1.33               h31b3aaa_0    conda-forge
llvm-openmp               10.0.0               hc9558a2_0    conda-forge
llvmlite                  0.32.0                   pypi_0    pypi
locket                    0.2.0                      py_2    conda-forge
lxml                      4.5.0                    pypi_0    pypi
lz4-c                     1.9.2                he1b5a44_1    conda-forge
make                      4.3                  h516909a_0    conda-forge
markupsafe                1.1.1            py38h1e0a361_1    conda-forge
matplotlib-base           3.2.1            py38h2af1d28_0    conda-forge
mccabe                    0.6.1                    pypi_0    pypi
more-itertools            8.2.0                    pypi_0    pypi
mpi                       1.0                       mpich    conda-forge
mpi4py                    3.0.3            py38h4a80816_1    conda-forge
mpich                     3.3.2                hc856adb_0    conda-forge
msgpack-python            1.0.0            py38hbf85e49_1    conda-forge
munch                     2.5.0                      py_0    conda-forge
nc-time-axis              1.2.0                      py_1    conda-forge
ncl                       6.6.2               h0f0234b_20    conda-forge
nco                       4.9.2           mpi_mpich_h9a76d41_102    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
netcdf-fortran            4.5.2           mpi_mpich_h6a79edc_4    conda-forge
netcdf4                   1.5.3           mpi_mpich_py38h894258e_3    conda-forge
networkx                  2.4                        py_1    conda-forge
numba                     0.49.0                   pypi_0    pypi
numpy                     1.18.4           py38h8854b6b_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openjpeg                  2.3.1                h981e76c_3    conda-forge
openssl                   1.1.1g               h516909a_0    conda-forge
ossuuid                   1.6.2             hf484d3e_1000    conda-forge
owslib                    0.19.2                     py_1    conda-forge
packaging                 20.1                       py_0    conda-forge
pandas                    1.0.3            py38hcb8c335_1    conda-forge
pango                     1.42.4               h7062337_4    conda-forge
partd                     1.1.0                      py_0    conda-forge
pathspec                  0.8.0                    pypi_0    pypi
pcre                      8.44                 he1b5a44_0    conda-forge
pcre2                     10.34                h2f06484_0    conda-forge
pep8-naming               0.4.1                    pypi_0    pypi
perl                      5.26.2            h516909a_1006    conda-forge
pillow                    7.1.2            py38h9776b28_0    conda-forge
pip                       20.1               pyh9f0ad1d_0    conda-forge
pixman                    0.38.0            h516909a_1003    conda-forge
pkg-config                0.29.2            h516909a_1006    conda-forge
pluggy                    0.13.1                   pypi_0    pypi
poppler                   0.67.0               h14e79db_8    conda-forge
poppler-data              0.4.9                         1    conda-forge
postgresql                12.2                 h8573dbc_1    conda-forge
progressbar2              3.51.3                   pypi_0    pypi
proj                      7.0.0                h966b41f_3    conda-forge
prospector                1.2.0                    pypi_0    pypi
prov                      1.5.3                    pypi_0    pypi
psutil                    5.7.0            py38h1e0a361_1    conda-forge
pthread-stubs             0.4               h14c3975_1001    conda-forge
py                        1.8.1                    pypi_0    pypi
pycodestyle               2.5.0                    pypi_0    pypi
pycparser                 2.20                       py_0    conda-forge
pydocstyle                5.0.2                    pypi_0    pypi
pydot                     1.4.1           py38h32f6830_1002    conda-forge
pyepsg                    0.4.0                      py_0    conda-forge
pyflakes                  2.1.1                    pypi_0    pypi
pygments                  2.6.1                    pypi_0    pypi
pykdtree                  1.3.1           py38h8790de6_1003    conda-forge
pyke                      1.1.1           py38h32f6830_1002    conda-forge
pylint                    2.4.4                    pypi_0    pypi
pylint-celery             0.3                      pypi_0    pypi
pylint-django             2.0.12                   pypi_0    pypi
pylint-flask              0.6                      pypi_0    pypi
pylint-plugin-utils       0.6                      pypi_0    pypi
pyopenssl                 19.1.0                     py_1    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyproj                    2.6.1.post1      py38h7521cb9_0    conda-forge
pyroma                    2.6                      pypi_0    pypi
pyshp                     2.1.0                      py_0    conda-forge
pysocks                   1.7.1            py38h32f6830_1    conda-forge
pytest                    5.4.1                    pypi_0    pypi
pytest-cov                2.8.1                    pypi_0    pypi
pytest-env                0.6.2                    pypi_0    pypi
pytest-flake8             1.0.5                    pypi_0    pypi
pytest-html               2.1.1                    pypi_0    pypi
pytest-metadata           1.9.0                    pypi_0    pypi
python                    3.8.2           he5300dc_7_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-stratify           0.1.1           py38h8790de6_1002    conda-forge
python-utils              2.4.0                    pypi_0    pypi
python_abi                3.8                      1_cp38    conda-forge
pytz                      2020.1             pyh9f0ad1d_0    conda-forge
pyyaml                    5.3.1            py38h1e0a361_0    conda-forge
r-base                    4.0.0                hdca8982_3    conda-forge
r-curl                    4.3               r40hcdcec82_1    conda-forge
r-udunits2                0.13            r40hcdcec82_1004    conda-forge
rdflib                    5.0.0            py38h32f6830_2    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
requests                  2.23.0             pyh8c360ce_2    conda-forge
requirements-detector     0.6                      pypi_0    pypi
rope                      0.17.0                   pypi_0    pypi
rstcheck                  3.3.1                    pypi_0    pypi
scikit-learn              0.22.2.post1     py38hcdab131_0    conda-forge
scipy                     1.4.1            py38h18bccfc_3    conda-forge
seaborn                   0.10.1                   pypi_0    pypi
seawater                  3.3.4                    pypi_0    pypi
sed                       4.7               h1bed415_1000    conda-forge
setoptconf                0.2.0                    pypi_0    pypi
setuptools                46.1.3           py38h32f6830_0    conda-forge
shapely                   1.7.0            py38hd168ffb_3    conda-forge
six                       1.14.0                     py_1    conda-forge
snowballstemmer           2.0.0                    pypi_0    pypi
sortedcontainers          2.1.0                      py_0    conda-forge
sparqlwrapper             1.8.5           py38h32f6830_1003    conda-forge
sphinx                    3.0.3                    pypi_0    pypi
sphinx-rtd-theme          0.4.3                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    1.0.3                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.4                    pypi_0    pypi
sqlite                    3.30.1               hcee41ef_0    conda-forge
tbb                       2018.0.5             h6bb024c_0  
tblib                     1.6.0                      py_0    conda-forge
tempest-remap             2.0.3           mpi_mpich_hf005093_8    conda-forge
termcolor                 1.1.0                    pypi_0    pypi
tiledb                    1.7.7                h8efa9f0_1    conda-forge
tk                        8.6.10               hed695b0_0    conda-forge
tktable                   2.10                 h555a92e_3    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.4            py38h1e0a361_1    conda-forge
tqdm                      4.46.0                   pypi_0    pypi
typing_extensions         3.7.4.2                    py_0    conda-forge
tzcode                    2020a                h516909a_0    conda-forge
udunits2                  2.2.27.6          h4e0c4b3_1001    conda-forge
urllib3                   1.25.9                     py_0    conda-forge
vmprof                    0.4.15                   pypi_0    pypi
wcwidth                   0.1.9                    pypi_0    pypi
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.34.2                     py_1    conda-forge
wrapt                     1.11.2                   pypi_0    pypi
xarray                    0.15.1                   pypi_0    pypi
xerces-c                  3.2.2             h8412b87_1004    conda-forge
xesmf                     0.3.0                    pypi_0    pypi
xlrd                      1.2.0                    pypi_0    pypi
xlsxwriter                1.2.8                    pypi_0    pypi
xorg-imake                1.0.7                         0    conda-forge
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.3             h84519dc_1000    conda-forge
xorg-libx11               1.6.9                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxaw               1.0.13            h14c3975_1002    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxmu               1.1.3                h516909a_0    conda-forge
xorg-libxpm               3.5.13               h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-libxt                1.1.5             h516909a_1003    conda-forge
xorg-makedepend           1.0.6                he1b5a44_1    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge
xz                        5.2.5                h516909a_0    conda-forge
yamale                    2.0.1                      py_0    conda-forge
yaml                      0.2.4                h516909a_0    conda-forge
yamllint                  1.23.0                   pypi_0    pypi
yapf                      0.30.0                   pypi_0    pypi
zict                      2.0.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h6597ccf_3    conda-forge
bascrezee commented 4 years ago

Can you try running with master (see commands in original post). What are the differences at the moment between master and v2.0.0?

bascrezee commented 4 years ago

For me the error also occurs for ESMValCore v2.0.0.

jvegreg commented 4 years ago

Can you try running with master (see commands in original post). What are the differences at the moment between master and v2.0.0?

I think they are almost identical. I used v2.0.0 because we are trying to test it as much at possible, not for any other thing.

Can you post the output of running conda list in your environment, just in case the error is appearing because of differences in the software stack

bascrezee commented 4 years ago

There are slight differences between our environments. I am building a new one now starting again from scratch.

bascrezee commented 4 years ago

So updating my environment solved this issue, at least for the CDS-SATELLITE-SOIL-MOISTURE dataset. I am quite surprised that such a major difference arises from a slightly different version of the Python packages. I am now doing more testing also for the other datasets, but it looks good so far.

jvegreg commented 4 years ago

I am quite surprised that such a major difference arises from a slightly different version of the Python packages.

I am not. It is quite easy to mess with the NaN values, and in that case you get 1e20 values all over the place and enormous errors. I am more worried when the differences are small, as it is usually more difficult to find the source of the error,

valeriupredoi commented 4 years ago

we should probably have a set of tests that test the machine errors - bitwise equality if of course impossible (even though we don't actually use compiled code) but differences in eg numpy or scipy versions may introduce small differences, I reckon a set of tests would be able to detect if those go over some threshold :beer: