search

teksi / wastewater

[DEV] Future TEKSI wastewater module, adapted data model to fit VSA-DSS 2020 new standard
https://teksi.github.io/wastewater
GNU General Public License v3.0
2 stars 5 forks source link

Optimize Import for large dataset #206

Open ponceta opened 6 months ago

ponceta commented 6 months ago

We want the user to be able to import large datasets from INTERLIS 2020_1 to TEKSI.

sjib commented 6 months ago

It takes very long when deselecting a class on the import wizard form (up to 5 or even 10 minutes). I can only deselect one class at a time 20240415_deselect_class_import

domi4484 commented 6 months ago

At the moment all to be imported objects are stored in a sql alchemy session which can be seen as a sort of transaction. When we deselect/select a class in the import dialog, all objects are removed/added to that session and this is a slow operation. The solution to that in my opinion is to rewrite the dialog and import procedure so that all objects are in memory only at the beginning. By accepting the dialog only the selected object/classes are added to the session and committed to the working schema.