finer concurrency on reformat

CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/

https://cadwrdeltamodeling.github.io/dms_datastore/

MIT License

1 stars 0 forks source link

finer concurrency on reformat #34

Open water-e opened 6 months ago

water-e commented 6 months ago

This is the limiter on my hardware. Takes 3 hours.

Currently the individual agencies are multi-processed which means 5 cores. We could do "third agencies" using file patterns, which are supported by the main routine ... will discuss with Nicky how this will affect Jenkins.