Openocd no graceful shutdown, blocked ports and conflicts

[snerzkristall]

The openocddriver opens and listens to multiple ports when started (except if last load command is shutdown) by the client. If the openocdserver/gdbserver is waiting too long for a connection (gdb, telnet, udp) the client throws an error (i assume timeout). This is not a problem on the client side, but the openocd server does not shut down correctly which leads to blocked ports or libusb access conflicts. The same problems occur when the client terminates the command that started openocd on the remote machine (via keyboard interrupt).

When openocd is started locally (w/o labgrid), it invokes the "shutdown" command upon receiving SIGTERM, so all ports are closed properly.

Is there any way to implement a graceful shutdown remotely?

This is very annoying because the only way to regain access to the ports and avoid libusb conflicts is restarting the remote machine (or at least the coordinator and exporter).

[Bastian-Krause]

So this boils down to "you use keyboard interrupts in your regular workflow which shuts down all things openocd gracefully normally, but not with labgrid", right? I'd like to understand your use case better. Could you explain why you use this regularly?

[snerzkristall]

In short, I am trying to set up a testbed for remote hardware debugging with multiple boards. Clients connect to the coordinator/exporter machine, acquire a place/board, flash new firmware onto the board and debug the new firmware on the remote board within an IDE (e.g. CLion).
So the client would start CLion, build the firmware, start a "labgrid client docker container", flash the new firmware onto the board (custom openocd driver) and start a gdb-server on the raspi, connect to the gdb-server and start debugging the firmwar eon the remote target.

When gdb disconnects properly the gdb-server/openocd is shut down. However if the connection times out or the connection is disconnected because of keyboard interrupts, i can still stop gdb on the local machine manually, but openocd does not shutdown properly.

I guess it is not a labgrid issue and more openocd, but maybe there is a way to force openocd shutdown upon openocd driver deactivation?

[Bastian-Krause]

I guess the openocd subprocess should be isolated from CTRL-C with subprocess.Popen(.., start_new_session=True), similar to the ssh keepalive process in #967 (/cc @jluebbe)?

[jluebbe]

Hmm, we should first find out why openocd is not shut down correctly. Perhaps it's disabling some signals?

If we use start_new_session=True, we'd need to handle the shutdown from labgrid in all cases, which doesn't seem easier that than the current state. ;)

Perhaps we can make sure that openocd terminates when its stdio pipes are closed?

[Bastian-Krause]

@snerzkristall Did you investigate this further?