spine-tools / Spine-Database-API

Database interface to Spine generic data model
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
6 stars 5 forks source link

More async fetching #205

Closed manuelma closed 1 year ago

manuelma commented 1 year ago

Introduces a lazy, light-weight ORM.

This PR introduces a new class called DBCache (a subclass of the native python dict) for holding the contents of a DB in memory. DBCache maps table names to ids to CacheItem instances, where CacheItem is also a subclass of dict.

The DBCache can be populated incrementally by querying the different tables and processing the results by chunks. CacheItem.__getitem__ is overriden to mimic a SQL JOIN, so for example users can access the field "class_name" for an object - which is not in the object table, but "class_id" is. So we use the class_id to find the object_class in the associated DBCache and then return its name. If the object_class is not yet in the cache, we advance the object_class query on demand until we find it. So it's as lazy as it gets.

Checklist before merging

codecov-commenter commented 1 year ago

Codecov Report

Base: 74.49% // Head: 74.53% // Increases project coverage by +0.04% :tada:

Coverage data is based on head (2ef743f) compared to base (2b45db6). Patch coverage: 74.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #205 +/- ## ========================================== + Coverage 74.49% 74.53% +0.04% ========================================== Files 76 77 +1 Lines 11789 12002 +213 Branches 2273 2347 +74 ========================================== + Hits 8782 8946 +164 - Misses 2560 2591 +31 - Partials 447 465 +18 ``` | [Impacted Files](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project) | Coverage Δ | | |---|---|---| | [spinedb\_api/filters/tools.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZmlsdGVycy90b29scy5weQ==) | `97.16% <ø> (ø)` | | | [spinedb\_api/export\_functions.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZXhwb3J0X2Z1bmN0aW9ucy5weQ==) | `85.33% <57.14%> (-2.34%)` | :arrow_down: | | [spinedb\_api/db\_mapping\_update\_mixin.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGJfbWFwcGluZ191cGRhdGVfbWl4aW4ucHk=) | `79.20% <64.28%> (+0.34%)` | :arrow_up: | | [spinedb\_api/db\_mapping\_base.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGJfbWFwcGluZ19iYXNlLnB5) | `87.97% <64.61%> (+4.62%)` | :arrow_up: | | [spinedb\_api/diff\_db\_mapping.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGlmZl9kYl9tYXBwaW5nLnB5) | `85.96% <65.21%> (-1.43%)` | :arrow_down: | | [spinedb\_api/db\_mapping\_commit\_mixin.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGJfbWFwcGluZ19jb21taXRfbWl4aW4ucHk=) | `75.00% <71.42%> (-6.09%)` | :arrow_down: | | [spinedb\_api/db\_mapping\_remove\_mixin.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGJfbWFwcGluZ19yZW1vdmVfbWl4aW4ucHk=) | `96.47% <71.42%> (-1.17%)` | :arrow_down: | | [spinedb\_api/db\_mapping\_add\_mixin.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGJfbWFwcGluZ19hZGRfbWl4aW4ucHk=) | `78.92% <75.00%> (-1.23%)` | :arrow_down: | | [spinedb\_api/db\_cache.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvZGJfY2FjaGUucHk=) | `75.06% <75.06%> (ø)` | | | [spinedb\_api/check\_functions.py](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project#diff-c3BpbmVkYl9hcGkvY2hlY2tfZnVuY3Rpb25zLnB5) | `61.60% <100.00%> (ø)` | | | ... and [7 more](https://codecov.io/gh/Spine-project/Spine-Database-API/pull/205?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project) | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Spine-project)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.