search

CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/
https://cadwrdeltamodeling.github.io/dms_datastore/
1 stars 0 forks source link

Sort out host-dependent speed issues #43

Open water-e opened 10 months ago

water-e commented 10 months ago

After running on the cloud, time needed for populating the repo seems to vary enormously when run on Azure and the Jenkins run may be yet another unknown. Is this web connectivity or proximity to Modeling_Data? Reformatting is very slow and there is no download associated with it.

water-e commented 10 months ago

On Azure, download is 2.5 hours while reformat is 5 hours. Think the reformat could be split up more in a very straightforward way (only a few hours work), but there is one prerequisite issue -- splitting up the logging better so that the work is concurrency safe. Will prioritize that.

water-e commented 10 months ago

That issue is https://github.com/CADWRDeltaModeling/dms_datastore/issues/35