search

observatoire-mobilite / odmkraken

The **kraken** is the orchestration layer responsible for gathering and postprocessing ODM's mobility data
MIT License
0 stars 0 forks source link

Import of raw data is slow #36

Closed gilgeorges closed 1 year ago

gilgeorges commented 1 year ago

During a test, where two icts_data import jobs ran in parallel, import times exploded - see last comment in #4: it took over an hour to read in a single file. Make sure this was due to the machine reaching resource limits, and not degraded query performance as the database grew to not that much data.

gilgeorges commented 1 year ago

Kicking this from release v0.1.0 as execution times aren't great, but acceptable