Data store spine DB and importer with excel

[pontusjn]

Hello,

I have found that there is an inconsistency with resetting the database back to a default empty file for data stores. One solution that I found is to delete the links in and out of said data store before doing so. Importing a template manually from a JSON-file to the database while having links also seem to not work. This is something I have to do very often because I am using an importer from an excel file which tends to not being able to overwrite certain types of inputs and I am changing values in the excel file frequently. Even after cleaning the data store database and running the importer I sometimes end up with the same database as the last time I ran the project with the old database values from the old excel file. My only solution to this issue so far is restarting spine, only for the issue to reappear again a couple of times later. Because of this happening only sometimes, I have to double check that my importer worked properly after each time the project is ran to see if everything got imported properly.

[jkiviluo]

Thanks for the question. Are you using purge command in the workflow arrow? And have the desired settings for the purge? If this is all correct, could you share an example where this goes wrong?

[pontusjn]

Thank you for a very quick response, I am very happy to see engagement in this project from so many parties. For context and possibly better understanding of the issue, I am a student working on a project similar to the tutorial https://spine-toolbox.readthedocs.io/en/latest/case_study_a5.html but with a different case study. My importer piggy-backs on how the existing one in the tutorial works, but with a heavily modified excel file because it is a different river. I had only seen the option purge when selecting the data store until now that you mentioned there's the option to auto-purge before writing. If it works as it should, that would save me a lot of time. I had not tried purging prior to now because I did not want anything to linger and I didn't know which items to check. Because of this my previous way of mass deletion was to create a new database file and overwriting the old one and then reapplying the template json-file I was using. When that became tedious, I switched to dragging and dropping a clean database file with the template already applied. I am going to try out "purge before writing" and see how it works before I can provide more feedback.

[jkiviluo]

Typically you want to use the 'select entity and value items' as well as 'select scenario items' to empty the database from 'just data'. Then things like entity classes, parameter definitions and icon graphics will remain.

[pontusjn]

Thank you for the clarification, I ended up trying with those settings for the purge and there was a clear case of old items still lingering. I ran the project once with an imported excel file that had 90% of the objects and relationships removed and they seemed to be gone at first but upon clicking on the "node"-tab in the object tree I could still view the removed parts in the entity graph and tabs no longer in use were still in bold letters even if they were empty. I am not sure whether or not these occurances has any effect on the optimisation-problem or not, but it would be hard to detect wrongly implemented changes.

I will test and make sure this bug is repeatable and come back to you. The temporary solution for me is to stick to removing links to the input datastore, drag and drop a clean database file with json-template in the directory and then reattach links.

[jkiviluo]

Looks like there is something going sideways. Needs to be checked --> a small case that reliably fails would be very nice.

[jkiviluo]

Maybe this issue does not exist anymore. Long time since the last problem.