Killing SliceRecon does not kill ZMQ

[schoonhovenrichard]

When trying to kill a SliceRecon process I run into several problems.

Firstly, I start 4 terminals as follows: Terminal 1) recast3d, Terminal 2) ./slicerecon_server --slice-size et.c Terminal 3) ./slicerecon_push_tomos --parallel (this one finishes the process on its own) Terminal 4) python slicerecon_plugin.py

I have not found a way to kill these processes without some "ZMQbg/1" process remaining alive.

Option A, killing from SliceVisualizer:

• If I hit Ctrl-D while in SliceVisualizer, it kills the graphics inside the screen and I get a "core dumped" in Terminal 2 that runs the slicerecon_server. Hitting Ctrl-C has no effect. It also does not close Terminals 1 and 4.
• Subsequently hitting Ctrl-C inside terminal 1 will kill SliveVisualizer but not the python process in Terminal 4.
• The python process in Terminal 4 can only be killed by Ctrl-Z. No error is thrown. However, subsequently trying to restart it leads to the error that the tcp address is already in use. If I run "ps" I see a process "ZMQbg/1" which needs to be killed by "kill -9 PID".

Option B, killing from terminal:

• If we navigate to Terminal 1 or Terminal 2 and hit Ctrl-C we can exit these terminals without error, in both cases the python process in Terminal 4 remains active. It needs to be killed with Ctrl-Z separately again. Again a "ZMQbg/1" process remains active that needs to be killed manually with "kill -9 PID".

I am unsure how the correct procedure to kill SliceRecon and the plugins should be implemented. Any thoughts?

PS: One reason why this is relevant is that when developing a python plugin, ideally we would keep the first 3 terminals active, see the result of the plugin, kill the pluging to make some changes en restart it again. I don't know if SliceRecon supports reconnecting a plugin in this way but it could probably be implemented if the visualization server has some loop which tries to reconnect. Currently, we can kill only the Python process in Terminal 4, run kill the "ZMQbg/1" but then restarting the python plugin does not work.

[jwbuurlage]

The system does support dynamically registering and unregistering plugins, so it is definitely possible to make a change to a plugin while running, and to kill/restart it.

IIRC, multithreaded Python scripts intercept the kill signal and don’t shut down.

Ctrl+Z does not kill a process, but just sends it to the background and pauses it. You can use jobs to see the current bg jobs (active and paused) and kill them with kill %x where x is the number shown in the jobs output. This should work for the plugin, and allow you to restart it.

After restarting it, trigger a slice reconstruction by clicking on a slice in RECAST3D, and you should see output for the new plugin.

There are three improvements to the code that can be done to make this use case more streamlined. It would be a good learning experience if anyone would like to try and tackle this.

1. Trigger reconstructions automatically after registering a (new) plugin.
2. Properly handle server closure by sending a SceneClosed packet through the entire pipeline. This has been on my list for a while. Is possibly a bit tricky because the order in which various stages are closed may matter.
3. Explicitly handle kill signals in the multithreaded python scripts to make them killable with ctrl+c, maybe as a helper function in the Tomopackets Python library.

I am also willing to work on making multi-scene and dynamic plugins a bit more streamlined (and to avoid crashing components of the pipeline because the endpoint disconnects without warning), but they are not a high priority. The dynamic plugins are only relevant for development so I can accept the manual refresh, killing Python scripts can be done with ctrl+Z followed by kill %, and while the crash is ugly we want to the software to close there anyway 🤷‍♂️.

— JW

[adriaangraas]

Does this problem still arise on develop? Curious if #1 fixed this. Points 1,2 are still very welcome though.