search

spine-tools / Spine-Database-API

Database interface to Spine generic data model
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
6 stars 5 forks source link

Reduce import_mappings memory usage #177

Closed soininen closed 2 years ago

soininen commented 2 years ago

It is possible to run out of memory when importing large datasets (see the discussion in Spine-project/Spine-Toolbox#1742). We should try to reduce the memory footprint of import_mappings.

soininen commented 2 years ago

The above commit removes most duplicates from the mapped_data data structure in import_mappings reducing the memory footprint by over 20% when importing 10000 objects with single parameter value into empty database without noticeable impact on import time.

soininen commented 2 years ago

I did some profiling on import_mappings and managed to bring down the time spent in get_mapped_data() by more than 50% in my test case.