ciandonovan
commented
1 year ago I'm working on resolving this myself, but can't figure out where the POSIX signal handlers are actually installed.
It looks like there are OS-agnostic handlers referenced around the line below, but the SIGTERM handler is never called because it doesn't install a POSIX handler for it, so the entire process gets automatically terminated by the kernel on receipt.
https://github.com/ros2/launch/blob/rolling/launch/launch/launch_service.py#L216
There is a POSIX signal handler for SIGINT installed somewhere, since its receipt doesn't terminate the process and instead correctly calls _on_sigint(signum).
The __handlers dict in signal_management.py stores mappings from signals to handler function addresses from what I can make out, but would appreciate help in identifying where the real handler is set at the OS level so I can stop a SIGTERM from killing everything.
@wjwwood mentioned in the TODO
fujitatomoya
Bug report
Required Info:
Steps to reproduce issue
Start a ROS2 Launch process which spawns at least one node, such as
Send a SIGTERM to the ROS2 launch process with
or
Use the official ROS2 OCI compliant containers from Docker Hub and run with Docker or Podman
And then stop the container with
Expected behavior
ROS2 Launch should catch the SIGTERM and propagate that signal to each of its child nodes.
Actual behavior
SIGTERM is ignored in the first instance, and in the second containerized example SIGTERM is still ignored but all processes are SIGKILL'd after the
podman stoptimeout.Additional information
The ROS2 Launch design article says
"if someone attempts to SIGTERM the launch system, they probably did so out of impatience after sending SIGINT to the launch system, and therefore the launch system should attempt to exit quickly."
This is not an accurate assumption.
The Linux signal man page defines, according to the POSIX.1-1990 standard, "SIGINT" as an "Interrupt from keyboard" and "SIGTERM" as a "Termination signal".
The GNU manual entry on termination signals goes further, defining "SIGINT" as being "sent when the user types the INTR character (normally C-c)" and "SIGTERM" as "a generic signal used to cause program termination".
Neither of these sources make reference to any hierarchy or precedence of these two signals.
Users of the ROS2 Launch system are likely to first run into this issue when try to automate the running of their ROS2 nodes on embedded systems, in systemd services, podman/docker containers, or both!
By default systemd services send SIGTERMs to their processes on shutdown, followed by SIGKILLs after a timeout.
Similarly, Podman/Docker containers get sent a SIGTERM to PID1 when running, for example,
podman stop ros_signal_test.The issue with containers cropped up here https://github.com/ros2/launch/issues/473, but it was only resolved for SIGINTs. An issue of SIGTERMs not being handled was mentioned here https://github.com/ros2/launch/issues/545#issuecomment-1191890187, but I've not been able to find any progress on it.
My current workaround for clean termination of an embedded ROS2 Launch system is to set
KillMode=SIGINTfor systemd services, or run Podman/Docker containers with the--stop-signal=SIGINTflag or setSTOPSIGNAL SIGINTin the Dockerfile. This only works when ROS2 Launch either detects it's running in non-interactive mode or that is set explicitly with--noninteractivehttps://github.com/ros2/launch/pull/475.However, it's oxymoronic to have to both be in "non-interactive" mode and to send a "keyboard interrupt" signal (SIGINT) for clean termination!
Feature request
Feature description
Have the ROS2 Launch system catch SIGTERMs and forward the same signal to their child nodes.
Implementation considerations
Windows??