Add coordinate fix for MIROC5

xtibau commented 5 years ago

Hi, I'm trying to get daily and monthly data from piControl of MIROC5 model. To do so, I'm using a recipe with no script.

authors:

pred_va

maintainer:
pred_va

references:
acknow_project

datasets:
- {dataset: MIROC5, project: CMIP5, mip: day, exp: piControl, ensemble: r1i1p1, start_year: 2101, end_year: 2600}
- {dataset: MIROC5, project: CMIP5, mip: Amon, exp: piControl, ensemble: r1i1p1, start_year: 2101, end_year: 2600}
diagnostics:

neurips: description: Load data for neurips challange. variables:
```
  clt: 
project: CMIP5
ensemble: r1i1p1
exp: piControl
start_year: 2101
end_year: 2600

  hfls: 
project: CMIP5
ensemble: r1i1p1
exp: piControl
start_year: 2101
end_year: 2600

scripts: null
```

My problem is that no data is saved in preproc folder. I found this in main_log.txt:

iris.exceptions.ConcatenateError: failed to concatenate into a single cube.
  An unexpected problem prevented concatenation.
  Expected only a single cube, found 2.

I'm using the latest version of ESMValTool installed with conda. Do you know what I am doing wrong?

mattiarighi commented 5 years ago

Can you try to run the Amon and day cases separately? And possibly also clt and hfls separately?

Can you also have a look at the debug logger, maybe there is some more information there.

xtibau commented 5 years ago

It doesn't work. I attach the run folder, for monthly data and clt var.

main_log.txt main_log_debug.txt recipe_MIROC5_part_1.txt

mattiarighi commented 5 years ago

Did you try also hfls? Same error?

Can you try to load and concatenate the files outside the tool, just on the command line?

clt_Amon_MIROC5_piControl_r1i1p1_210001-219912.nc clt_Amon_MIROC5_piControl_r1i1p1_220001-229912.nc clt_Amon_MIROC5_piControl_r1i1p1_230001-239912.nc clt_Amon_MIROC5_piControl_r1i1p1_240001-249912.nc clt_Amon_MIROC5_piControl_r1i1p1_250001-259912.nc clt_Amon_MIROC5_piControl_r1i1p1_260001-266912.nc

I'll try to reproduce it using the recipe you posted.

xtibau commented 5 years ago

I got the same error for hfls. I tried directly on the command line with cdo and seems to work.

mattiarighi commented 5 years ago

cdo is not an issue, the problem is reading the files with Iris.

Trying this in the command line, it looks like the problem is in the data:

>>> iris.cube.CubeList(cubes).concatenate_cube()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/lustre01/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/iris/cube.py", line 502, in concatenate_cube
    check_aux_coords=check_aux_coords)
  File "/mnt/lustre01/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/iris/_concatenate.py", line 272, in concatenate
    check_aux_coords)
  File "/mnt/lustre01/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/iris/_concatenate.py", line 718, in register
    match = self._cube_signature.match(cube_signature, error_on_mismatch)
  File "/mnt/lustre01/pf/b/b309057/SOFTWARE/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/iris/_concatenate.py", line 476, in match
    raise iris.exceptions.ConcatenateError(msgs)
iris.exceptions.ConcatenateError: failed to concatenate into a single cube.
  Cube metadata differs for phenomenon: cloud_area_fraction

mattiarighi commented 5 years ago

@valeriupredoi has lots of experience with this concatenation stuff...

valeriupredoi commented 5 years ago

detailed response follows concatenation is prevented due to some differences in the cube.metadata.attributes eg differences between first two cubes' metadata attrs:

tracking_id: 979a9496-7256-48c6-9079-9f5dc38c4a4f / 8c59969d-e245-4852-8e54-2c90de6f2961
creation_date: 2011-09-15T07:03:42Z / 2011-09-15T06:57:10Z
history: 2011-09-15T07:03:42Z altered by CMOR: Converted units from '' to '%'. /2011-09-15T07:03:42Z altered by CMOR: replaced missing value flag (-999) with standard missing value (1e+20). 2011-09-15T07:03:42Z altered by CMOR: Inverted axis: lat. 2011-09-15T06:57:10Z altered by CMOR: Converted units from '' to '%'. 2011-09-15T06:57:10Z altered by CMOR: replaced missing value flag (-999) with standard missing value (1e+20). 2011-09-15T06:57:10Z altered by CMOR: Inverted axis: lat.

there is, however, a function that fixes all these attributes before concatenation (and I have just checked and after applying the function, the attributes and the whole metadata is identical across the seven cubes).

Now, raw concatenation should produce a list of cubes with a single member (not that concatenate() is the raw concatenation and concatenate_cube() is the wrapper that extracts the single element or fails if there are more elements in that list). And, not surprisingly, the raw concatenation spits out a two-element list:

[<iris 'Cube' of cloud_area_fraction / (%) (time: 6840; latitude: 128; longitude: 256)>,
<iris 'Cube' of cloud_area_fraction / (%) (time: 1200; latitude: 128; longitude: 256)>]

that, after further examination, proves to be the list of all cubes bar the fifth cube, and of course the fifth cube (the bugger, the fifth cube is clt_Amon_MIROC5_piControl_r1i1p1_240001-249912.nc)

Now, looking at any give cube and comparing it with the 5th one seems to give us identical results but the thing to remember is iris is most picky and refuses concatenation when even the slights difference in coordinates. Of course, time points can differ but anything else should be absolutely identical across the cubes to concatenate. A simple comparison of the latitude coordinate reveals that a few points differ, very very small differences but they differ eg:

In [138]: c1.coord("latitude").points[63]                                                                                                  
Out[138]: -0.7003838029733238

In [139]: c5.coord("latitude").points[63]                                                                                                  
Out[139]: -0.7003838029733237

this blocks concatenation! Once we assign:

c5.coord("latitude").points = c1.coord("latitude").points
c5.coord("latitude").bounds = c1.coord("latitude").bounds

then concatenation works fine:

cubeY = iris.cube.CubeList([c1,c2,c3,c4,c5,c6,c7]).concatenate_cube()

Now, the differences in coordinate points are a fault of either the cmorization process or the model itself (if it outputs in cmorized netCDF format). How to fix it? Add a callback function before the call to cconcatenate that checks for these issues

schlunma commented 5 years ago

We have a similar case (tas for this model), where we simply round the coordinates. See here.

Since apprently multiple variables are affected, we could implement this fix for all variables (using AllVar).

mattiarighi commented 5 years ago

Thank you @valeriupredoi and @schlunma !

@xtibau do you want to try implementing the fix (in the core)?

xtibau commented 5 years ago

Thanks a lot @schlunma @mattiarighi and @valeriupredoi.

I'll try to fix it!

valeriupredoi commented 5 years ago

Rounding the coords values is a very good idea, no reason they should be that precise. Lemme know if you need help with the fix :beer:

Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

On Tue, 6 Aug 2019, 10:14 xtibau, notifications@github.com wrote:

Thanks a lot @schlunma https://github.com/schlunma @mattiarighi https://github.com/mattiarighi and @valeriupredoi https://github.com/valeriupredoi.

I'll try to fix it!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESMValGroup/ESMValCore/issues/198?email_source=notifications&email_token=AG5EFI3ODTXJZUAB2OL5DUDQDE6I3A5CNFSM4IJS264KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3UPQWQ#issuecomment-518584410, or mute the thread https://github.com/notifications/unsubscribe-auth/AG5EFI3BOLFXIECTM5DMYO3QDE6I3ANCNFSM4IJS264A .

valeriupredoi commented 5 years ago

oh and cheers @schlunma for pointing out the prior case :beer:

ESMValGroup / ESMValCore

Add coordinate fix for MIROC5 #198