search

spine-tools / Spine-Toolbox

Spine Toolbox is an open source Python package to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases.
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
69 stars 17 forks source link

Frequently unable to commit changes to DB #1884

Open DillonJ opened 1 year ago

DillonJ commented 1 year ago

Very often, after working on a SpineDB and I go to commit the changes, it doesn't work. It happens often after interrupting a workflow execution but I'm not sure if that is always the case. I have the below errors in the console.

This has happened alot over the lifetime of toolbox, where I have a current SpineDB with a lot of changes to it and I can't commit them and am forced to redo them all again. I suppose it's been for many different reasons. Is there anyway we can salvage a db with changes that haven't been committed yet? Like write some sort of recovery file or something like that? Some fallback?

C:\Workspace\Spine>cd c:\workspace\spine\Spine-Toolbox

c:\Workspace\Spine\Spine-Toolbox>python spinetoolbox.py
Exception in thread Thread-79:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\returning_process.py", line 97, in _do_listen
    one_shot_process_semaphore.release()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 66, in release
    raise RuntimeError("Logic error: process counter negative.")
RuntimeError: Logic error: process counter negative.
Exception in thread Thread-120:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\returning_process.py", line 97, in _do_listen
    one_shot_process_semaphore.release()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 66, in release
    raise RuntimeError("Logic error: process counter negative.")
RuntimeError: Logic error: process counter negative.
Exception in thread Thread-118:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\spine_engine.py", line 549, in _execute_item_filtered
    item_finish_state = item.execute(filtered_forward_resources, filtered_backward_resources)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-items\spine_items\merger\executable_item.py", line 66, in execute
    return_value = self._process.run_until_complete()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\returning_process.py", line 46, in run_until_complete
    return return_value
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 71, in __exit__
    self.release()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 66, in release
    raise RuntimeError("Logic error: process counter negative.")
RuntimeError: Logic error: process counter negative.
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundObjectParameterDefinitionModel(0x2351cecba60) at 0x0000023512781840> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518c31010, name="MainWindow") at 0x0000023512A82F40> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundRelationshipParameterDefinitionModel(0x2351cecb740) at 0x0000023512781800> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518c31010, name="MainWindow") at 0x0000023512A82F40> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\metadata_table_model_base.py", line 115, in canFetchMore
    result |= self._db_mngr.can_fetch_more(db_map, fetch_parent, listener=self._db_editor)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.fetch_parent.FlexibleFetchParent object at 0x0000023512F28AC0> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518c31010, name="MainWindow") at 0x0000023512A82F40> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\metadata_table_model_base.py", line 115, in canFetchMore
    result |= self._db_mngr.can_fetch_more(db_map, fetch_parent, listener=self._db_editor)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.fetch_parent.FlexibleFetchParent object at 0x0000023512F28910> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518c31010, name="MainWindow") at 0x0000023512A82F40> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundObjectParameterValueModel(0x2351cecbc90) at 0x0000023512E038C0> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518c31010, name="MainWindow") at 0x0000023512A82F40> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundRelationshipParameterValueModel(0x2351cecbb00) at 0x00000235127816C0> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518c31010, name="MainWindow") at 0x0000023512A82F40> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Exception in thread Thread-218:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\returning_process.py", line 97, in _do_listen
    one_shot_process_semaphore.release()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 66, in release
    raise RuntimeError("Logic error: process counter negative.")
RuntimeError: Logic error: process counter negative.
QAbstractItemModel::endRemoveRows:  Invalid index ( 8 , 4 ) in model CompoundObjectParameterValueModel(0x2351e3c9eb0)
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 54005)
Traceback (most recent call last):
  File "C:\Python\Python38\lib\socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Python\Python38\lib\socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "C:\Python\Python38\lib\socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python\Python38\lib\socketserver.py", line 747, in __init__
    self.handle()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spinedb-api\spinedb_api\spine_db_server.py", line 526, in handle
    self.request.sendall(response + bytes(self._EOT, self._ENCODING))
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
----------------------------------------
Exception in thread Thread-215:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\spine_engine.py", line 549, in _execute_item_filtered
    item_finish_state = item.execute(filtered_forward_resources, filtered_backward_resources)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-items\spine_items\merger\executable_item.py", line 66, in execute
    return_value = self._process.run_until_complete()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\returning_process.py", line 46, in run_until_complete
    return return_value
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 71, in __exit__
    self.release()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 66, in release
    raise RuntimeError("Logic error: process counter negative.")
RuntimeError: Logic error: process counter negative.
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundObjectParameterDefinitionModel(0x2351e3c93c0) at 0x000002351663CB00> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518416eb0, name="MainWindow") at 0x0000023516614780> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundRelationshipParameterDefinitionModel(0x2351e3ca180) at 0x000002351663CAC0> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518416eb0, name="MainWindow") at 0x0000023516614780> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\metadata_table_model_base.py", line 115, in canFetchMore
    result |= self._db_mngr.can_fetch_more(db_map, fetch_parent, listener=self._db_editor)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.fetch_parent.FlexibleFetchParent object at 0x0000023512F89F70> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518416eb0, name="MainWindow") at 0x0000023516614780> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\metadata_table_model_base.py", line 115, in canFetchMore
    result |= self._db_mngr.can_fetch_more(db_map, fetch_parent, listener=self._db_editor)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.fetch_parent.FlexibleFetchParent object at 0x0000023512F89610> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518416eb0, name="MainWindow") at 0x0000023516614780> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundObjectParameterValueModel(0x2351e3c9eb0) at 0x000002351663C540> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518416eb0, name="MainWindow") at 0x0000023516614780> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Traceback (most recent call last):
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in canFetchMore
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_editor\mvcmodels\compound_parameter_models.py", line 70, in <genexpr>
    return any(self.db_mngr.can_fetch_more(db_map, self, listener=self._parent) for db_map in self.db_maps)
  File "c:\Workspace\Spine\Spine-Toolbox\spinetoolbox\spine_db_manager.py", line 248, in can_fetch_more
    raise RuntimeError(
RuntimeError: can't associate parent <spinetoolbox.spine_db_editor.mvcmodels.compound_parameter_models.CompoundRelationshipParameterValueModel(0x2351e3c9050) at 0x000002351663C840> to listener <spinetoolbox.spine_db_editor.widgets.spine_db_editor.SpineDBEditor(0x23518416eb0, name="MainWindow") at 0x0000023516614780> - not listening on DB map <spinedb_api.diff_db_mapping.DiffDatabaseMapping object at 0x000002351815DE80>
Exception in thread Thread-319:
Traceback (most recent call last):
  File "C:\Python\Python38\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Python\Python38\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\returning_process.py", line 97, in _do_listen
    one_shot_process_semaphore.release()
  File "C:\Workspace\Spine\Spine-Toolbox\src\spine-engine\spine_engine\utils\execution_resources.py", line 66, in release
    raise RuntimeError("Logic error: process counter negative.")
RuntimeError: Logic error: process counter negative.
manuelma commented 1 year ago

It doesn't work as in the UI doesn't respond? You are unable to show the commit dialog? You commit via the dialog but changes do not stick in the DB?

(you have a lot of errors in those tracebacks that are probably related to a lot of toolbox activity (removing things in db editor, stoping tools, etc)

manuelma commented 1 year ago

I'm not a huge fan of implementing a recovery path that could also have its own issues - seems better to me to spend the effort in fixing the commit itself. Anyways, we do have some sort of recovery path which is Hamburger -> Export to create another sqlite or json file that then could be reimported - have you tried that?

DillonJ commented 1 year ago

The dialog opens - I type a message press ok and nothing happens - it fails silently and the asterix beside the db name doesn't go away

jkiviluo commented 1 year ago

Maybe related: https://github.com/irena-flextool/flextool/issues/32