CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/
https://cadwrdeltamodeling.github.io/dms_datastore/
MIT License
1 stars 0 forks source link

Add argument to populate repo to allow for partial year update #21

Open dwr-psandhu opened 7 months ago

dwr-psandhu commented 7 months ago

Description

Currently the populate scripts downloads the entire period of record available for each station.

Change

Adding an argument that allows for updating only for a partial window (resolution could be year or decade) will greatly speed up the daily update and also lower the burden for the data providers services.

dwr-psandhu commented 6 months ago

This just needs a flag to skip populate_repo 1 and 2 and go to populate_repo step 3, i.e. year 2000 and beyond. This will greatly increase the update speed (to minutes from hours) with the understanding that data providers rarely update past decade data. If they do, the entire repo can be rebuilt with not specifying the flag.

water-e commented 6 months ago

The work you delineated sounds right and the trial implementation is fine, but that was never the main concern: