Parallel processing - Githubissues

rcgsheffield / urban_flows

Research Computing work for the Urban Flows project

2 stars 2 forks source link

Open Joe-Heffer-Shef opened 3 years ago

Joe-Heffer-Shef commented 3 years ago

Enable multi-threading to sync in parallel and increase throughput?

For Awesome data bridge: run AQI and data in separate processes? so they both sync in near real-time

Joe-Heffer-Shef commented 3 years ago

Use an ETL framework (e.g. Apache Airflow) so all harvesters can use concurrency