Healthcheck fails on OpenResty

[erseldev]

I've been actively using version 3.5.5 for CrowdSec bouncers (firewall bouncer and django bouncer for django apps). However I wasn't able to implement nginx bouncer due to lack of lua package in version 3.5.5. So, I switched to 4.0.0-openresty-alpine-fat this weekend, however I couldn't get it running in a healthy state.

Healthcheck fails and the container stays in unhealthy condition. Pls find below the Health log from inspect:

----------------------------------------
"Health": {
                "Status": "unhealthy",
                "FailingStreak": 4,
                "Log": [
                    {
                        "Start": "2024-03-17T21:37:51.899899623+03:00",
                        "End": "2024-03-17T21:37:51.962438883+03:00",
                        "ExitCode": 7,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-17T21:38:21.967048446+03:00",
                        "End": "2024-03-17T21:38:22.016149571+03:00",
                        "ExitCode": 7,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-17T21:38:52.019909614+03:00",
                        "End": "2024-03-17T21:38:52.076661981+03:00",
                        "ExitCode": 7,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-17T21:39:22.080164811+03:00",
                        "End": "2024-03-17T21:39:22.130057679+03:00",
                        "ExitCode": 7,
                        "Output": ""
                    }
                ]
            }

------------------------------------

I have no clue what I'm doing wrong?

[theseion]

Thanks for reporting. We'll look into, but it may take us a couple of days.

@TafkaMax, could you take a look?

[TafkaMax]

OK, will check once I have time. It seems the new version of CRS dropped. I have to test that out aswel...

What does docker logs <openresty-modsec-crs-container-name> say about the container ?

[erseldev]

Hi, The log is as follows:

-------------------
ted@ash1:~$ docker logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/conf.d/default.conf.template to /usr/local/openresty/nginx/conf/conf.d/default.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/conf.d/logging.conf.template to /usr/local/openresty/nginx/conf/conf.d/logging.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/conf.d/modsecurity.conf.template to /usr/local/openresty/nginx/conf/conf.d/modsecurity.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/nginx.conf.template to /usr/local/openresty/nginx/conf/nginx.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/modsecurity.d/modsecurity-override.conf.template to /usr/local/openresty/nginx/conf/modsecurity.d/modsecurity-override.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/modsecurity.d/setup.conf.template to /usr/local/openresty/nginx/conf/modsecurity.d/setup.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/includes/proxy_backend_ssl.conf.template to /usr/local/openresty/nginx/conf/includes/proxy_backend_ssl.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/includes/proxy_backend.conf.template to /usr/local/openresty/nginx/conf/includes/proxy_backend.conf
20-envsubst-on-templates.sh: Running envsubst on /usr/local/openresty/nginx/templates/includes/location_common.conf.template to /usr/local/openresty/nginx/conf/includes/location_common.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/25-listen-on-ipv6-by-default.sh
25-listen-on-ipv6-by-default.sh: info: Getting the checksum of /usr/local/openresty/nginx/conf/conf.d/default.conf
25-listen-on-ipv6-by-default.sh: info: /usr/local/openresty/nginx/conf/conf.d/default.conf differs from the packaged version
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/90-copy-modsecurity-config.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/91-update-resolver.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/92-update-real_ip.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/94-activate-plugins.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/95-activate-rules.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/03/18 18:32:48 [notice] 1#1: ModSecurity-nginx v1.0.3 (rules loaded inline/local/remote: 0/797/0)
ted@ash1:~$

--------------------

This may not be so meaningful so I'm also posting the terminal messages:

--------------------
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/90-copy-modsecurity-config.sh
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/91-update-resolver.sh
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/92-update-real_ip.sh
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/94-activate-plugins.sh
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/95-activate-rules.sh
nginx        | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx        | 2024/03/18 18:32:48 [notice] 1#1: ModSecurity-nginx v1.0.3 (rules loaded inline/local/remote: 0/797/0)
Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container nginx is unhealthy
ted@ash1:~/docker/proxy$
--------------------

I made a brief comparison between 3.5.5 and 4.0.0 and figured that generate-certificate script is not listed in "/usr/local/bin/" folder for 4.0.0 (healthcheck script is there). Could this be the reason?

[erseldev]

Let me also add this:

ted@ash1:~/docker/proxy$ docker exec -it nginx curl http://localhost:80/healthz
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>openresty</center>
</body>
</html>
ted@ash1:~/docker/proxy$

[theseion]

Thanks @erseldev. The missing certificates aren't good but shouldn't be the reason for the issue. However, it looks like there's no endpoint for the health check.