search

observatoire-mobilite / odmkraken

The **kraken** is the orchestration layer responsible for gathering and postprocessing ODM's mobility data
MIT License
0 stars 0 forks source link

Use dagster-dask for distributed computing #37

Open gilgeorges opened 1 year ago

gilgeorges commented 1 year ago

Especially the small jobs like extract_halts might benefit from using Dask - might also fix #36

gilgeorges commented 1 year ago

Usefulness of dask remains to be seen - for now, execution seems more than fast enough (even though exporting the job description data may be quite an overhead)