spine-tools / Spine-Toolbox

Spine Toolbox is an open source Python package to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases.
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
69 stars 17 forks source link

Entity_alternative table freezes/very slow with a large copy-paste #2867

Open jkiviluo opened 2 months ago

jkiviluo commented 2 months ago

When copy pasting e.g. 2000 entities to the entity_alternative table (with alternative_name and active status), Toolbox freezes.

Copy pasting 2000 entities in the "add entities" dialog on the other hand is fast even though it also sets the active status.

Originally mentioned in https://github.com/spine-tools/Spine-Toolbox/issues/2864

jkiviluo commented 2 months ago

Actually, it did go through in the end, it just took a long time.

Furthermore, there seems to be some kind of limit with 1000 entities - anything beyond that was not displayed in the entity_alternative table after adding 2000 entities through 'add entity' dialog.

e-zaline commented 2 months ago

Btw when we have a large number or entities in the model, even a medium-sized copy-paste (100 elements) takes quite a long time to go through

PiispaH commented 2 months ago

Made some changes in the above PR. The one thing I found was that when adding data into the tables, toolbox tries to autofill the entity_class_name column. This autofilling becomes slower when the size of the database grows. I disabled the autofilling for rows where the class is defined when data is pasted into the tables. This means that toolbox no longer tries to correct the entity class name, so users should be sure that their data is correct before they paste it to the tables.

On my test database the time it took to paste and add the entities was about 20 times faster than previously, but still twice as long as through Add entities. Because of that keeping this issue open.

PiispaH commented 1 month ago

I tried to write a benchmark for the table pasting here. I ran into some weird issues where during the running of the benchmark, often a sqlite3 error stating that a table named list_value was not found in the database would stop the run. Because the benchmark runs the test function multiple times, a miracle is needed for it to pass all the 15 or so test rounds for the benchmark to complete. If the benchmark worked, the pasting could be accurately profiled with ease by utilizing pyperfs profiling features.

The branch contains the scraps of the benchmark, some test data and a couple of "temporary" unit tests that are just there to figure out how to write the benchmarks. The other test that adds the entities through the add entities dialog is also not working properly for some reason.