ESMValGroup / ESMValCore

ESMValCore: A community tool for pre-processing data from Earth system models in CMIP and running analysis scripts.

https://www.esmvaltool.org

Apache License 2.0

42 stars 38 forks source link

What happens when fx_files are needed but not found? #103

Closed ledm closed 4 years ago

ledm commented 5 years ago

At the moment, some of the preprocessors use fx_files (like masking, the area/volume statistics and the upcoming zonal/meridional statitistics).

It's not obvious what should happen when fx_files are absent. In the case of model data, I think that the preprocessor should fail without fx_files.

In the case of observational data, if fx_files are absent, then the tool should be able to estimate the grid's area or volume.

This is complicated by the fact that the statistics tools can apply several operators:

The mean and the sum operations require the area/volume fields.
The sum operations can sometimes require the area/volume fields and sometimes doesn't need it.
The minimum, maximum operations do not require the area/volume.
The standard deviation, variance and median operators in iris are not weighted aggregators, but should require the area/volume.

It's a bit confusing!

valeriupredoi commented 5 years ago

the fallback option for preprocessor function (mask landsea) when no fx files are available or they're broken (sometimes they are!) is to use Natural Earth shapefiles (for masking land at a higher resolution and seas at a lower resolution, but both resolutions much higher than the highest model resolution ie 10m and 50m << 1km). This option is straightforward and precise (even more precise than the fx files themselves, which may not necessarily be a good thing though) BUT it's not implemented for irregular grids and will spit a ValueError if masking with NE files is attempted on irregular grids.

Can you not use the masking functions before doing any statistical operations on the data? Or you actually need the fx files in your diagnostic to manipulate different ratios and stuff

ledm commented 5 years ago

The area_statistics and volume_statistics operators are independent of the masking preprocessor.

In order to calculate an average over the data, we need to weight the mean by the size of the grid cells. We can't use the natural earth shape files to describe the grid area or volume as they are grid specific.

valeriupredoi commented 5 years ago

ok, makes sense, dunno really, not sure what can be done if the fx files are not there at all, do however see the comment https://github.com/ESMValGroup/ESMValCore/issues/24#issuecomment-503613912 since that will take care of about 90% of the problem :grin:

valeriupredoi commented 4 years ago

230 #297 #170 should have fixed this; please reopen if still in the bogs :beer: