There are scenarios in which the sidecar handles stopping / restarting it's forked collectors poorly:
The collector is a wrapper script that forks additional processes
The wrapper script runs the collector with elevated privileges (sudo)
This PR tries to improve handling this as much as possible.
Instead of killing just the forked PID, it also resorts to killing the entire process group.
This should cover case 1
For case 2 the sidecar can only avoid waiting forever on a process it is not allowed to kill. (lack of permissions)
This is fixed by adjusting the timeout loop and introducing a terminate channel to abort the goroutine that is endlessly wait(2)ing.
The timeout between sending a SIGTERM and SIGKILL can be adjusted with a new configration option:
collector_shutdown_timeout: "10s"
In addition:
improve the status of the backands reporting to Graylog
The GetRotatedLog() writers were closed immideatly after creating them.
This had no effect and can be removed.
Don't prevent sidecar from shutting down with hanging collectors
There are scenarios in which the sidecar handles stopping / restarting it's forked collectors poorly:
This PR tries to improve handling this as much as possible.
Instead of killing just the forked PID, it also resorts to killing the entire process group. This should cover case
1
For case
2
the sidecar can only avoid waiting forever on a process it is not allowed to kill. (lack of permissions) This is fixed by adjusting the timeout loop and introducing a terminate channel to abort the goroutine that is endlessly wait(2)ing.The timeout between sending a
SIGTERM
andSIGKILL
can be adjusted with a new configration option:collector_shutdown_timeout: "10s"
In addition: