metno / pyaerocom

Python tools for climate and air quality model evaluation
https://pyaerocom.readthedocs.io/
GNU General Public License v3.0
25 stars 13 forks source link

Model maps shown for EMEP should use the frequency of the Base_*.nc file rather than the colocated data and then resampling #1260

Closed lewisblake closed 1 week ago

lewisblake commented 3 months ago

Is your feature request related to a problem? Please describe. An inefficiency in the creation of the model maps is that for some models, such as EMEP, we already have yearly files available, so reading in daily data and then temporally averaging is unnecessarily expensive.

Describe the solution you would like to see In the case of making maps for the EMEP model, the maps_freq argument set in a config file should be used to read the Base_{maps_freq}.nc file already available, regardless of the colocation frequency main_freq. Some information in the logger should be present to inform that this option is being used.

Additional context Blocked by #1259

heikoklein commented 3 months ago

This is critical for the trends-evaluation, which does collocation in monthly data which takes 45min for 4 periods without model-maps. Model-maps for one period takes 6h alone, since daily data is read.

dulte commented 2 months ago

I've started on the problem in #1312. If I understand the problem correctly, this should solve it, even though it is not in the most elegant way:

I'm testing out simply changing main_freq to maps_freq when the maps are calculated(or the coarsest freq if that option is chosen). It of course then colocates the obs wrong. It seems that ModelMapsEngine creates a colocator, but only uses the model data, in which case the obs are irrelevant. And as I understand it, the obs shown on the map are created elsewhere, so this does not matter anyway(?). The obs shown will also be just the July data for that year, and not a mean of the year.

The freq change is done for all models, no special case for EMEP. I can change that to check if the model is EMEP, as we are not always sure to have yearly data

I think my code is a bit misplaced, but due to the overuse of OOP, it is not easy to know where things are happening. But we can discuss that later