search

ietf-tools / www

A customized CMS for the IETF website
BSD 3-Clause "New" or "Revised" License
22 stars 45 forks source link

Logging: Redirect stdout from child processes to Supervisor #432

Closed microamp closed 2 months ago

microamp commented 2 months ago

Currently, only Supervisor logs would appear when we run

kubectl logs -f www-wagtail-...
2024-04-22 23:57:48,162 INFO supervisord started with pid 1
2024-04-22 23:57:49,164 INFO spawned: 'gunicorn' with pid 7
2024-04-22 23:57:49,165 INFO spawned: 'nginx' with pid 8
2024-04-22 23:57:50,282 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 23:57:50,282 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
...

which is consistent with what we are seeing in Grafana.

2024-04-20 18:46:22,662 INFO supervisord started with pid 1
2024-04-20 18:46:23,665 INFO spawned: 'gunicorn' with pid 6
2024-04-20 18:46:23,666 INFO spawned: 'nginx' with pid 7
2024-04-20 18:46:24,884 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-20 18:46:24,884 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
...

It would be very useful to be able to view all logs from the child processes in Grafana as well.

Consider redirecting logs from Nginx and Gunicorn processes to Supervisor so that they can be picked up by the Grafana agent. One way to do that is to explicitly set the log file to /dev/stdout with log rotation disabled.

/dev/stdout is a symlink to /proc/self/fd/1. When a process opens that file, the system actually clones file descriptor #１ (stdout) of that process. Using this as stdout_logfile therefore causes supervisord to redirect the program’s stdout to its own stdout (from where it will be captured by Docker)

https://veithen.io/2015/01/08/supervisord-redirecting-stdout.html