Limited precision of ERA5 data

[axel-lauer]

Limited precision of ERA5 data

@ESMValGroup/scientific-lead-development-team @ESMValGroup/technical-lead-development-team

ERA5 netcdf data (e.g. downloaded with era5cli) seem to have a precision problem. An example of the problem is given below (monthly mean specific humidity at 100 hPa in January 2020).

While this is probably not a big problem at lower levels, the limited precision is problematic for e.g. stratospheric analyses. Additional testing suggests, that only the cached ("preprocessed") netcdf files from the CDS are affected. If reprocessing is triggered via the CDS interface (e.g. by selecting only a single level), the netcdf output looks fine.

A possible solution for this problem could be to extend the ERA5 fix to be able to process the original grib files instead of the cached (preprocessed) netcdf files downloaded by era5cli. As ERA5 data are important for many studies, I propose to add finding a solution for this problem to our mid-term strategy (https://github.com/ESMValGroup/ESMValTool/discussions/3035).

Data downloaded from CDS in grib format

Data downloaded with era5cli in netcdf format (similar to data downloaded from CDS in netcdf format)

[katjaweigel]

@axel-lauer Thanks for pointing to this issue! It would be great to be able to process grib directly, also because DKRZ dowloads these files: https://docs.dkrz.de/doc/dataservices/finding_and_accessing_data/era_data/index.html#era-data

[zklaus]

Ouch. Good catch, @axel-lauer! I agree that it would be nice to have support for grib, both ERA5 specifically, and all kinds of grib in general. Moreover, it shouldn't be too hard since courtesy of iris-grib iris can already read grib. Some work around translating parameter numbers to CMIP variables may be required.

However, isn't this also just a data failure on the center's part? Have you reported it upstream?

[hb326]

@zklaus: we did indeed communicate this to ECMWF via their ticket system. We got the following answer: "Please note that the recommendation is to download the ERA5 data in its GRIB native format. The conversion to NetCDF format is provided for convenience but is experimental in the sense that the GRIB to NetCDF conversion should be ok for most cases but for some complicated GRIB products/mix of variables, there were issues such as the time coordinates not being correctly converted."

It seems like they will look into this at some point, but it is not a high priority for them.

[valeriupredoi]

However, isn't this also just a data failure on the center's part? Have you reported it upstream?

Spot on! I too think it's a processing error at CDS-server side. Apparently you can talk to a Knowledge Duck to raise an issue https://www.ecmwf.int/en/newsletter/169/news/climate-data-store-virtual-assistant :duck:

[zklaus]

Hmhm. Some convenience that is.

[valeriupredoi]

@zklaus: we did indeed communicate this to ECMWF via their ticket system. We got the following answer: "Please note that the recommendation is to download the ERA5 data in its GRIB native format. The conversion to NetCDF format is provided for convenience but is experimental in the sense that the GRIB to NetCDF conversion should be ok for most cases but for some complicated GRIB products/mix of variables, there were issues such as the time coordinates not being correctly converted."

It seems like they will look into this at some point, but it is not a high priority for them.

what a load of baloney! It either works or not, if it don't, you retire it and make it work, and deploy again. Can we use a fix for this once it enters esmvaltool? Or we're producing statistical noise by upping the precision artificially?

[valeriupredoi]

@hb326 what ticketing system did you use to get that rubbish answer? I might give them a piece of my mind too - actually, am curious what GRIB -> netCDF4 converter they use

[hb326]

@valeriupredoi: they use a JIRA system for their tickets. A colleague from our sister department actually opened the ticket and had me added for information purposes.

[remi-kazeroni]

@axel-lauer Thanks for pointing to this issue! It would be great to be able to process grib directly, also because DKRZ dowloads these files: https://docs.dkrz.de/doc/dataservices/finding_and_accessing_data/era_data/index.html#era-data

related to https://github.com/ESMValGroup/ESMValCore/issues/1991

[valeriupredoi]

@valeriupredoi: they use a JIRA system for their tickets. A colleague from our sister department actually opened the ticket and had me added for information purposes.

cheers, Birgit! :beer:

[FranziskaWinterstein]

Thank you for raising this issue here @axel-lauer . Since I am one of those 'suffering' from this error, I am interested in finding a near term solution or option for our analysis. We currently think about converting the GRIB files manually to netcdf and hopefully being able to pursue our analysis with that. If reading in GRIB files directly is more on the midterm schedule, would netcdf Files being helpful for you?

[FranziskaWinterstein]

Another comment: The GRIB Files provided by the DKRZ (which @katjaweigel mentioned) are on a different grid CDS: 1440x721 DKRZ: 1280x640. Do you think this is an issue, or which one would you prefer?