DOMjudge / domjudge-packaging

DOMjudge packaging for (Linux) distributions and live image
31 stars 37 forks source link

Add HEALTHCHECK to domserver Docker container #121

Closed edomora97 closed 2 years ago

edomora97 commented 2 years ago

When running the Docker container for domserver, it is possible that one of nginx or php-fpm crashes/stops working. When this happens from outside the container the effects can be quite mysterious. Adding an health check allows the user to diagnose more quickly what happened.

Running docker ps, in the column STATUS it is shown: Up X minutes (healthy) or Up X minutes (unhealthy) depending on the exit code of the health check script. To further diagnose the problem one can run docker inspect domserver (optionally piped to jq '.[0].State.Health') and this will report diagnostic information like:

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2021-12-19T19:22:53.555071655+01:00",
      "End": "2021-12-19T19:22:54.012109556+01:00",
      "ExitCode": 0,
      "Output": "nginx ok | php ok | http ok"
    },
    ...
  ]
}

This health check probes supervisord for the status of the two services (nginx and php-fpm), as well as tries to reach the API with an HTTP request. To perform this HTTP request I've used php -r since in the image there isn't curl nor wget. My PHP skills are crap, I won't be surprised if there's a better way to do that.

This commit also fixes the deprecated MAINTAINER labels in the various Dockerfiles.


Some context: we were running DJ in the docker container while suddenly the server switched off (probably due to a power loss). When it got back on the docker container restarted automatically but php-fpm failed to start logging:

[14-Dec-2021 14:49:05] ERROR: Another FPM instance seems to already listen on /var/run/php-fpm-domjudge.sock
[14-Dec-2021 14:49:05] ERROR: FPM initialization failed

From the outside the result was a 502 from "nginx", but from which nginx was not clear since we also had nginx running outside docker. And we were a bit confused by the fact that the container was still running fine (or at least we initially thought). We could diagnose the problem much more quickly if docker ps showed us that the container was not healthy. Why php-fpm couldn't start automatically is still a mystery to me, since that file (at the moment at least) does not exist. I'll try to further investigate this later.

edomora97 commented 2 years ago

What about adding curl, wget or httpie?

We can do that, I just tried to avoid adding another package to the container (especially for this simple task). But yeah, I will feel a bit safer if no PHP from me is present anywhere in the codebase 😂

Also there is an option since a few weeks to specify the path where domjudge is served from, we should take into account I think?

I didn't know about that, I'll check this immediately.

edomora97 commented 2 years ago

Also there is an option since a few weeks to specify the path where domjudge is served from, we should take into account I think?

I didn't know about that, I'll check this immediately.

Ok, I have two possible solutions for that:

  1. Use $WEBAPP_BASEURL inside the health check, but the value is the same provided to the container (the fixes done here are not present). So it is possible that something would break (even though I can't imagine why)
  2. Extract the base url from the generated configuration file (/opt/domjudge/domserver/webapp/config/static.yaml), which is written from here

I'm leaning towards option 2 to minimize the risk of doing something wrong or unexpected. What do you think?

The result would be something like:

BASEURL=$(grep domjudge.baseurl /opt/domjudge/domserver/webapp/config/static.yaml | awk '{ print $2 }')
if curl --silent --fail "$BASEURL/api" >/dev/null; then
    printf "http ok"
else
    printf "http error"
    ret=1
fi
nickygerritsen commented 2 years ago

Second option sounds good. And regarding the package: I think curl or wget are so small it’s totally fine

nickygerritsen commented 2 years ago

I also pushed a new 7.3.4 image to Docker Hub

Dup4 commented 2 years ago

image

It looks good to me.