search

Graylog2 / fpm-recipes

Graylog package build recipes
http://docs.graylog.org/en/latest/pages/installation/operating_system_packages.html
18 stars 19 forks source link

systemd unit file configuration conflicts with "graceful shutdown" #73

Closed tokred closed 7 years ago

tokred commented 7 years ago

Expected Behavior

My Graylog nodes are running behind a load balancer. For maintenance, I would like to use the "Graceful shutdown" feature (http://docs.graylog.org/en/2.1/pages/configuration/load_balancers.html#graceful-shutdown) e.g. to perform system upgrades. Expectation is that the Graylog service shuts down and the node is not active anymore.

Side question: Is there a difference between shutting Graylog down via web interface or directly via CLI shutdown, e.g. by machine reboot or by systemctl stop?

Current Behavior

After Graylog gracefully shuts down, systemd watchdog automatically restarts the service due to the unit files's configuration. This does not make any sense...

Log entries in journalctl confirm:

... systemd[1]: graylog-server.service holdoff time over, scheduling restart.
... systemd[1]: Starting Graylog server...

Possible Solution

Change the unit file's configuration parameter Restart=always to something more sane.

Steps to Reproduce (for bugs)

  1. Install Graylog using the official RPM and set up.
  2. Launch Graylog service.
  3. Stop a node via "Nodes" - "More actions" - "Graceful shutdown" and watch it going offline.
  4. Wait 20-30sec for the node to reappear and verify with systemctl status graylog-server.service.

    Context

Your Environment

bernd commented 7 years ago

@tokred Thank you for the report! :smiley: After reading up on the systemd Restart option, using on-failure is indeed a better value to use here.

Regarding your question:

Side question: Is there a difference between shutting Graylog down via web interface or directly via CLI shutdown, e.g. by machine reboot or by systemctl stop?

No, that's no difference. Stopping Graylog via systemd or a regular SIGTERM initiates a graceful shutdown.

bernd commented 7 years ago

@tokred I fixed it for the upcoming 2.2 release packages and for the 2.1 packages.

The graylog-server-2.1.1-2.noarch.rpm package should appear soon!