Graylog2 / collector-sidecar

Manage log collectors through Graylog
https://www.graylog.org/
Other
268 stars 56 forks source link

Improve process shutdown handling #462

Closed mpfz0r closed 1 year ago

mpfz0r commented 1 year ago

There are scenarios in which the sidecar handles stopping / restarting it's forked collectors poorly:

  1. The collector is a wrapper script that forks additional processes
  2. The wrapper script runs the collector with elevated privileges (sudo)

This PR tries to improve handling this as much as possible.

Instead of killing just the forked PID, it also resorts to killing the entire process group. This should cover case 1

For case 2 the sidecar can only avoid waiting forever on a process it is not allowed to kill. (lack of permissions) This is fixed by adjusting the timeout loop and introducing a terminate channel to abort the goroutine that is endlessly wait(2)ing.

The timeout between sending a SIGTERM and SIGKILL can be adjusted with a new configration option: collector_shutdown_timeout: "10s"

In addition:

danotorrey commented 1 year ago

Fixes #463

mpfz0r commented 1 year ago

@bernd @thll I think I've addressed all your comments. Thank you :+1: