Correct uwsgi health check in cluster (livenessProbe, readinessProbe)

[bittner]

I'm trying to come up with a suitable health check (for a OpenShift/Kubernetes livenessProbe and readinessProbe) for uWSGI. The configuration I'm using is:

# FILE: uwsgi.ini
[uwsgi]
master = true
processes = 4
threads = 2
buffer-size = 65535
socket = 0.0.0.0:8000
pidfile = /run/uwsgi-myapp.pid
wsgi-file = /app/myapp/wsgi.py

What I want to detect is when my Django application fails, hence a case like this:

$ uwsgi deployment/application/uwsgi.ini                                                                                                                                                                                                       
[uWSGI] getting INI configuration from deployment/application/uwsgi.ini                                                                                                                                                                                                         
*** Starting uWSGI 2.0.17.1 (64bit) on [Fri Dec 21 16:55:54 2018] ***                                                                                                                                                                                                           
compiled with version: 6.3.0 20170516 on 21 December 2018 16:53:29                                                                                                                                                                                                              
os: Linux-4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018                                                                                                                                                                                                         
  ...
*** Operational MODE: preforking+threaded ***                                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                                                                              
  File "/usr/local/lib/python3.7/site-packages/environ/environ.py", line 273, in get_value                                                                                                                                                                                      
    value = self.ENVIRON[var]                                                                                                                                                                                                                                                   
  File "/usr/local/lib/python3.7/os.py", line 678, in __getitem__                                                                                                                                                                                                               
    raise KeyError(key) from None                                                                                                                                                                                                                                               
KeyError: 'DJANGO_SECRET_KEY'                                                                                                                                                                                                                                                   
  ...
*** no app loaded. going in full dynamic mode ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 8)
spawned uWSGI worker 1 (pid: 9, cores: 2)
spawned uWSGI worker 2 (pid: 10, cores: 2)
spawned uWSGI worker 3 (pid: 12, cores: 2)
spawned uWSGI worker 4 (pid: 14, cores: 2)

Python exits due to some exception, but uwsgi keeps running. How can I detect this?

I found some hints on cluster health checks in the docs, but I'm not quite sure how to put that into practice. :worried:

Any hints?

[unbit]

Hi, i think you can use the --need-app flag to force uWSGI to exit if no app is loaded (will be the default in 2.1 as dynamic apps will be disabled)

[bittner]

When is 2.1 planned to be released?

[che0]

I think you can use either raw tcp connect to uwsgi port, or include https://github.com/m-messiah/uwget in the package and run that on some known endpoint and parse the result.

edit: I forked it to uwping, which checks for specific return code and you can install into your image and then use as Kubernetes liveness/readiness probe.

[jgirdner]

Here is what we use.

       readinessProbe:
          httpGet:
            path: /health/
            port: 9040
          initialDelaySeconds: 6
          timeoutSeconds: 1

We have https enabled and had to make an exception for /health/

SECURE_SSL_REDIRECT = True
# For GKE Health Check
SECURE_REDIRECT_EXEMPT = [
    '^health/',
]

[petrprikryl]

I am using https://github.com/andreif/uwsgi-tools

healthcheck:
  test: ["CMD-SHELL", "uwsgi_curl -H 'X_FORWARDED_PROTO: https' localhost:8000 ${HOST}/healthcheck/"]
  interval: 20s
  timeout: 10s
  retries: 15
  start_period: 30s