Bulk commit to DB and Cache IDs

[simonhkswan]

Inserting data into the database was quite slow, especially when we were commiting each individual value into the database in a diff-correlation metric. For every value we were also querying the database for a dataset_id and a metric_id.

So to speed up the process we now use a single session.commit() for each of the DataFrame metrics. We also use cachetools on the database/metric id functions. It's much faster locally and likely and even better speedup when the database is remote.

[codecov[bot]]

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (68fbadb) 79.9% compared to head (edb0d2e) 79.0%.

:exclamation: Current head edb0d2e differs from pull request most recent head 0be0676. Consider uploading reports for the commit 0be0676 to get more accurate results

Additional details and impacted files

[](https://app.codecov.io/gh/synthesized-io/insight/pull/162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=synthesized-io) ```diff @@ Coverage Diff @@ ## master #162 +/- ## ======================================== - Coverage 79.9% 79.0% -0.9% ======================================== Files 11 11 Lines 926 956 +30 Branches 145 155 +10 ======================================== + Hits 740 756 +16 - Misses 142 149 +7 - Partials 44 51 +7 ``` | [Files](https://app.codecov.io/gh/synthesized-io/insight/pull/162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=synthesized-io) | Coverage Δ | | |---|---|---| | [src/insight/metrics/metrics\_usage.py](https://app.codecov.io/gh/synthesized-io/insight/pull/162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=synthesized-io#diff-c3JjL2luc2lnaHQvbWV0cmljcy9tZXRyaWNzX3VzYWdlLnB5) | `94.1% <100.0%> (ø)` | | | [src/insight/metrics/base.py](https://app.codecov.io/gh/synthesized-io/insight/pull/162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=synthesized-io#diff-c3JjL2luc2lnaHQvbWV0cmljcy9iYXNlLnB5) | `72.1% <93.7%> (-2.7%)` | :arrow_down: | | [src/insight/database/utils.py](https://app.codecov.io/gh/synthesized-io/insight/pull/162?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=synthesized-io#diff-c3JjL2luc2lnaHQvZGF0YWJhc2UvdXRpbHMucHk=) | `73.1% <71.4%> (-3.2%)` | :arrow_down: |

[sonarcloud[bot]]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
75.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud