Open dwr-psandhu opened 7 months ago
This just needs a flag to skip populate_repo 1 and 2 and go to populate_repo step 3, i.e. year 2000 and beyond. This will greatly increase the update speed (to minutes from hours) with the understanding that data providers rarely update past decade data. If they do, the entire repo can be rebuilt with not specifying the flag.
The work you delineated sounds right and the trial implementation is fine, but that was never the main concern:
Description
Currently the populate scripts downloads the entire period of record available for each station.
Change
Adding an argument that allows for updating only for a partial window (resolution could be year or decade) will greatly speed up the daily update and also lower the burden for the data providers services.