search

CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/
https://cadwrdeltamodeling.github.io/dms_datastore/
MIT License
1 stars 0 forks source link

Sort out host-dependent speed issues #43

Open water-e opened 5 months ago

water-e commented 5 months ago

After running on the cloud, time needed for populating the repo seems to vary enormously when run on Azure and the Jenkins run may be yet another unknown. Is this web connectivity or proximity to Modeling_Data? Reformatting is very slow and there is no download associated with it.

water-e commented 5 months ago

On Azure, download is 2.5 hours while reformat is 5 hours. Think the reformat could be split up more in a very straightforward way (only a few hours work), but there is one prerequisite issue -- splitting up the logging better so that the work is concurrency safe. Will prioritize that.

water-e commented 5 months ago

That issue is https://github.com/CADWRDeltaModeling/dms_datastore/issues/35