docker-entrypoint.sh does not forward termination signals to Java process

[tklovett]

I've been playing around with Graylog running in Docker on Kubernetes (via the official helm chart), and I've found that Graylog pods take a long time to terminate, then eventually terminate without logging any graceful shutdown messages. That seemed suspicious to me, and I wondered if the eventual termination was only due to the terminationGracePeriodSeconds: 120.

If I'm not mistaken, the problem boils down to Kubernetes being unable to get its SIGTERM all the way to the Java process in the pod. Kubernetes sends a SIGTERM to the process with PID 1, which is docker-entrypoint.sh, but the script does not forward it to the java process. When the grace period expires, Kubernetes sends an SIGKILL to all processes in the pod, achieving the ungraceful termination.

Steps to Reproduce:

docker run --name graylog graylog/graylog:3.0.2-2
# it'll log tons of Mongo & Elasticseacrch connection errors, but that's fine
docker exec -it graylog sh -c 'kill -TERM 1'
# java process continues running, docker container continues running
docker exec -it graylog sh -c 'kill -TERM $(pidof java)'
# java process terminates, docker container stops

docker run --name graylog graylog/graylog:3.0.2-2
docker kill --signal TERM graylog
# signal sent successfully, but container is still running
docker kill --signal KILL graylog
# container terminates ungracefully

Solutions

1. Modify docker-entrypoint.sh (PID 1) to trap and forward SIGTERM to the java child process
2. Use a specialized init system, like tini, like so:

# Add Tini
ENV TINI_VERSION <check-version-on-github>
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini
ENTRYPOINT ["/tini", "--"]
# Run your program under Tini
CMD ["/docker-entrypoint.sh"]

https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-terminating-with-grace https://cloud.google.com/solutions/best-practices-for-building-containers#signal-handling

[tklovett]

This may be why @KongZ created this custom pre-stop hook for the pods in kubes:

    lifecycle:
      preStop:
        exec:
          command:
          - bash
          - -ec
          - |
            ROOT_PASSWORD=`/k8s/kubectl get secret graylog -o "jsonpath={.data['graylog-password-secret']}" | base64 -d`
            curl -XPOST -sS -u "admin:${ROOT_PASSWORD}" "localhost:9000/api/system/shutdown/shutdown"```

[KongZ]

@tklovett that's correct. the POST /api/system/shutdown will call System.exit(0). More over the shutdown will also flush all messages on inputs and stop receiving messages.

[jalogisch]

this is fixed in #88