We are running consul agent using consul docker image. Docker image is build using the consul image. When container is stopped graceful exit is happening but we can see in the logs that a second sigterm is received by the consul agent. Not sure from where it's coming.
From the docker we are executing a shell script start.sh which runs consul agent command in background. In consul agent configuration we have enabled
Since the consul agent command is running as a child process the sigterm won't be forwarded to consul agent process, so we have trapped the sigterm in start.sh and forward it to consul process.
We are assuming that consul process will get the SIGTERM and perform a graceful shutdown.. In the logs however its logging a second SIGTERM which we are not sure from where its coming.
{"@level":"info","@message":"Caught second signal, Exiting","@module":"agent","@timestamp":"2022-05-05T18:08:44.327411Z","signal":15}
We even tried running docker consul image with docker compose and 3 nodes and faced the same issue.
Snippet of docker-compose.yml
Overview of the Issue
Consul agent is getting a Second sigterm signal.
We are running consul agent using consul docker image. Docker image is build using the consul image. When container is stopped graceful exit is happening but we can see in the logs that a second sigterm is received by the consul agent. Not sure from where it's coming.
From the docker we are executing a shell script
start.sh
which runs consul agent command in background. In consul agent configuration we have enabledfor the graceful shutdown.
Since the consul agent command is running as a child process the sigterm won't be forwarded to consul agent process, so we have trapped the sigterm in start.sh and forward it to consul process.
We are assuming that consul process will get the SIGTERM and perform a graceful shutdown.. In the logs however its logging a second SIGTERM which we are not sure from where its coming.
We even tried running docker consul image with docker compose and 3 nodes and faced the same issue. Snippet of docker-compose.yml
server config
If we ommit the
leave_on_terminate
andskip_leave_on_interrupt
than second sigterm is not recieved, but graceful exit won't happen.With
leave_on_terminate
Without
leave_on_terminate
{"@level":"warn","@message":"serf: Shutdown without a Leave","@module":"agent.server.serf.lan","@timestamp":"2022-05-10T05:33:23.098134Z"}