Importer fails occasionally

[jkiviluo]

This happens sometimes: I try to import more than one scenario into the Toolbox. The first import attempt gets stuck (some of the scenarios are almost complete, i.e. only the green final rows are missing, but the rest of the scenarios do not start). I press Stop. Everything looks ok in Toolbox UI. No error messages in console. I'll try to run the importer again. Spine Toolbox freezes. When I forcibly close the Toolbox, I get this:

(spi38) C:\data\SpineToolbox>python spinetoolbox.py
--------------------------------------------------------------------------------
Exception happened during processing of request from ('127.0.0.1', 57696)
(spi38) C:\data\SpineToolbox>Exception happened during processing of request from ('127.0.0.1', 57694)
Traceback (most recent call last):
Traceback (most recent call last):  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 316, in _handle_request_noblock    self.process_request(request, client_address)  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 316, in _handle_request_noblock    self.process_request(request, client_address)  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 347, in process_request    self.finish_request(request, client_address)  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 360, in finish_request    self.RequestHandlerClass(request, client_address, self)  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 347, in process_request    self.finish_request(request, client_address)  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 747, in __init__    self.handle()  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 360, in finish_request    self.RequestHandlerClass(request, client_address, self)  File "C:\data\SpineToolbox\src\spinedb-api\spinedb_api\spine_db_server.py", line 502, in handle    self.request.sendall(response + bytes(self._EOT, self._ENCODING))
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\socketserver.py", line 747, in __init__    self.handle()
----------------------------------------  File "C:\data\SpineToolbox\src\spinedb-api\spinedb_api\spine_db_server.py", line 502, in handle    self.request.sendall(response + bytes(self._EOT, self._ENCODING))
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
----------------------------------------
Process Process-2:
Traceback (most recent call last):  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\multiprocessing\process.py", line 315, in _bootstrap    self.run()  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\multiprocessing\process.py", line 108, in run    self._target(*self._args, **self._kwargs)  File "C:\data\SpineToolbox\src\spinedb-api\spinedb_api\spine_db_server.py", line 112, in _do_work    conn.send(result)  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\multiprocessing\connection.py", line 206, in send    self._send_bytes(_ForkingPickler.dumps(obj))  File "C:\Users\prokjt\Miniconda3\envs\spi38\lib\multiprocessing\connection.py", line 280, in _send_bytes    ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
BrokenPipeError: [WinError 232] The pipe is being closed

By the way, when the importer is working normally, it seems each scenario is imported fully (green letter messages appear), before it starts on the next scenario.

Also, this was GDX importer (not sure if happens with others). Also, the importers work if I just import one-by-one or e.g. two at once. When I do e.g. 5 scenarios at once it fails half the time.

[manuelma]

By the way, when the importer is working normally, it seems each scenario is imported fully (green letter messages appear), before it starts on the next scenario.

That's correct, shouldn't it be that way?

[jkiviluo]

That's correct, shouldn't it be that way?

Yes. Just reporting the normal behavior, so you know that issue is not in the data or importer specs.

[jkiviluo]

This may also be caused by 'purge' (doing it from the data store properties prior to importing).

[jkiviluo]

This keeps happening. Now it failed possibly with this (not sure when that traceback had appeared):

Traceback (most recent call last): File "C:\data\SpineToolbox\spinetoolbox\project_item\specification_editor_window.py", line 224, in closeEvent if not self.tear_down(): File "C:\data\SpineToolbox\src\spine-items\spine_items\exporter\widgets\specification_editor_window.py", line 1004, in tear_down self._preview_updater.tear_down() File "C:\data\SpineToolbox\src\spine-items\spine_items\exporter\widgets\preview_updater.py", line 460, in tear_down self._url_model.modelReset.disconnect(self._enable_controls) RuntimeError: Internal C++ object (FullUrlListModel) already deleted.

[manuelma]

I try to import more than one scenario into the Toolbox

What does this mean exactly? Do you have an importer specification and gdx file you can share? Or workflow? I don't understand what you mean by import scenarios, is the importer running in parallel? Where do the scenarios come from?

[soininen]

This keeps happening. Now it failed possibly with this (not sure when that traceback had appeared):

Looks like the traceback is originating from Exporter. I've reported it as #1870

[manuelma]

@jkiviluo So how do I reproduce it, I need a tool running parallel scenarios that feed to an importer than then feeds to a DB?

[jkiviluo]

At least I can reproduce something quite regularly: I open an importer open that is slow to load (there is some data series data in the gdx file it's trying to comprehend). I try to close it right away. I open another importer spec. Try to close it too. Keep doing this and usually Toolbox crashes quite soon.

[manuelma]

I see, thanks for that. Although that looks like a different issue? I'm still trying to understand the original one about the parallel importer execution - how to reproduce - if you could share a picture of a workflow even and briefly describe what's going on I could be able to figure it out.

[jkiviluo]

Ok, I built a small new project to test this, but I can't make it fail. It may then have something to do with the TradeRES Toolbox project. Among other things, it has MySQL connection (that I don't keep open).

I made another issue about that slowness: https://github.com/Spine-project/Spine-Toolbox/issues/1874