search

metno / pyaerocom

Python tools for climate and air quality model evaluation
https://pyaerocom.readthedocs.io/
GNU General Public License v3.0
24 stars 13 forks source link

pyaerocom usage of pyaro too slow #1302

Open dulte opened 1 month ago

dulte commented 1 month ago

The pyaro reader uses loops to convert pyaro data structures to ungriddeddata. This has worked fine for the data used by pyaro until now. But with EEA, where you have millions samples of samples per species per months, this method is slow. Reader needs to be numpyfied

dulte commented 1 month ago

Pm10 for one month is >5000 files, each file (should be) a single station... I also need to check if this is the case, or if there is overlap between files for some reason

lewisblake commented 1 month ago

From pyaerocom meeting: This is related to the caching. We should talk about this tomorrow at the Design Retreat. Previously we had explicitly not had caching in pyaro.

dulte commented 1 month ago

Not only. It is also slow because of the way I made it convert between pyaro and ungriddeddata. I foresaw this then I made the reader. So the time has come to do anything about it. Or is it the other issue you are talking about?