search

Deltares-research / kenmerkendewaarden

Derive indicators from waterlevel measurements
https://deltares-research.github.io/kenmerkendewaarden/
GNU General Public License v3.0
2 stars 0 forks source link

Speed up data retrieval process by using yearly freq #124

Closed veenstrajelmer closed 1 month ago

veenstrajelmer commented 2 months ago

With the many duplicates documented in https://github.com/Rijkswaterstaat/wm-ws-dl/issues/39, it is not always possible to retrieve timeseries per year. The maximum allowed of values in a returned timeseries by the DDL is 157681 and because of duplicates some year timeseries were larger. However, many duplicates were cleaned by now, so it should be possible to retrieve these without issues now. First check this by retrieving all measurements amounts again.

This will probably be somewhat faster, although longer timeseries also take longer to retrieve from DDL. When retrieving monthly timeseries, all 1848 iterations take approximately 1.5 seconds, totalling to approximately 40 minutes. When retrieving yearly timeseries, the empty or low-coverage years take 1.5 seconds, but the 10-minute years might take up to 20 seconds per iterations. For HOEKVHLD this totals to 17 minutes. All timings on WCF, seems to be significantly faster on laptop.

veenstrajelmer commented 1 month ago

Too tricky to depend on this for now. Already with DONAR, and we do not know how performant WADAR will be.