Concatenation of ocean 3D data (from two experiments) doesn't work if aux_coords differ

[valeriupredoi]

@koldunovn brought this up in a long-forgotten but alas still open #753 and I have just found this thanks to @npgillett asking me something offline - in a nutshell: concatenation of cubes at io.concatenate() stage gets completely hosed if cubes have aux_coords that differ (could be name, metadata, or actual data of one or more aux_coords - I have seen all of these manifest themselves), this is happening mostly for two different experiments that need to be used for data complementary (ie: historical from 1850 to 2014 and ssp45: from 2015 to the end of world due to Greta devouring it). Options we have to fix this bug/add enhancement to concatenation:

• ask @bjlittle and the iris folk to be less strict about cube.concatenate_cube()
• fix the aux_coords before concatenation - but this is tricky, which one of them would be the reference by which the other gets fixed?
• remove the aux_coords and re-add them to the resulting concatenated cube (that will concatenate this time around), but here, again, which one to be used as correct/reference?

Brainstorming ahoy! Also, if this has been resolved elsewhere I will buy a pint, but am sure it hasn't :beer:

[valeriupredoi]

also, here's an example recipe that @npgillett had issues with, perfectly showing the issue with the aux coords data differing:

---
documentation:
  description: |
    Produces figures from Gillett et al. (2020)
    'Constraining human contributions to observed warming since preindustrial'
  authors:
    - predoi_valeriu
  references:
    - gillett20ncc
preprocessors:
  regrid_5_5:
    regrid:
      target_grid: 5x5
      scheme: linear
diagnostics:
  gillett20_figs:
    description: Preprocess data needed for Gillett et al. (2020) figures.
    variables:
      siconc:
        preprocessor: regrid_5_5
        reference_dataset: HadCRUT4
        field: T2Ms
        mip: SImon
        project: CMIP6
        start_year: 1850
        end_year: 2019
        additional_datasets:
          - {dataset: FGOALS-g3, ensemble: r1i1p1f1, grid: gn, exp: [historical,ssp245] }
    scripts:
      fig1:
        script: examples/correlate.py

[npgillett]

Thanks @valeriupredoi for raising this here. Based on your comment that it was a difference in auxiliary coordinates, I narrowed this down to spurious zeroes in the auxiliary coordinates latitude and longitude in the historical siconc files from FGOALS-g3. It would be good if this didn't crash esmvaltool, but in the mean time I have emailed the modelling centre to ask them to republish the historical files with corrected auxiliary coordinates. I'll let you know when I hear back...

[valeriupredoi]

yep - I saw those zeroes and was quite puzzled as to why they were there, cheers for telling the modellers to fix it! Unfortunately it's not esmvaltool that's not concatenating because of that, it's iris that is complaining (and for a good reason here, actually!) - this case should be easy to fix then, but I am still a bit unsure how to approach the generalized issue :beer:

[jvegreg]

@sloosvel and I also had this issue in #944

[valeriupredoi]

so did you fix it there in any way @jvegasbsc ? :beer:

[jvegreg]

By brute force: we set up manually lat and lon values taken from an arbitrary file, as differences were in the order of 1e-6. Weirdly enough, we couldn't use the rounding trick because there were a couple of cases in which even rounding to 1e-2 or 1e-3 will not avoid the differences

[bouweandela]

spurious zeroes in the auxiliary coordinates latitude and longitude in the historical siconc files from FGOALS-g3.. I have emailed the modelling centre to ask them to republish the historical files with corrected auxiliary coordinates

Would it be possible to implement a dataset fix for this in the ESMValCore so it just works until the new data is available?

[bouweandela]

ask @bjlittle and the iris folk to be less strict about cube.concatenate_cube()

@valeriupredoi If this is not a data problem, it might be best to open an issue for this discussion in the iris repo, that way it will probably get a lot more attention.

[jvegreg]

ask @bjlittle and the iris folk to be less strict about cube.concatenate_cube()

Or at least to get the error to be more informative. It tool us a while to realize what was the real problem