nginx gives 504 gateway timeout

[jby]

Describe the bug If I run my docker container without debug, it gives me a 504 gateway timeout when trying to access the main page, wether from login or from within the application. If I run it in debug mode it works as expected, but painfully slow. I am able to access pages within the app if I address them directly in my browser. I.e: http://sal.company.tld/dashboard/1/

To Reproduce Steps to reproduce the behavior: Restart docker container without debug go to main page login timeout after some 50 seconds with error: 504 Gateway Time-out nginx/1.10.3 Expected behavior Promptly being logged in and able to use the app Screenshots See here: https://macadmins.slack.com/archives/C061B9XGS/p1612780384006500

Server (please complete the following information):

• Host environment RHEL 7.5.1804, Docker 1.13.1, docker.io/macadmins/sal

Client (please complete the following information):

• OS: macOS 11.2.1
• Browser Safari 14.0.3, Firefox 85.0.2

Additional context I used to run my munki-repo on this host as well as sal in docker - these problems started when I moved the munki-repo and disabled the host-nginx, re-enabling the host-nginx does not solve the problem however. If possible, please replicate the issue while Sal is in debug mode. In debug-mode it runs fine, albeit painfully slow

[grahamgilbert]

As discussed on slack, this is down to server resources. On or more of cpu, memory or disk speed on one or both of your app server or the database. This isn’t an issue with Sal really. You should monitor your resource consumption on both of your app servers and database servers and make changes accordingly. Your hosting platform may even give you insights as to any queries that may be running slowly (such as aws performance insights with rds).

[jby]

This is an on-prem RHEL host running under libvirtd. I'll try to monitor it's resource usage to see if I can find any bottlenecks.

[jby]

So, a summary from my DBA after doing monitoring of the timeout:

• longest running statement in the PG-DB took 70ms
• Some 6000 queries in total
• Total execution time of the above: 3.7 seconds
• Browser hangs for 30-60 seconds, then responds with the 504 timeout from nginx

I have a hard time to see how the DB is to blame for this.

• The server (libvirtd VM) is equipped with 4 CPUs and 16GB RAM
• It's virtually impossible to measure how little of these resources are utilised during the hang/timeout The only thing visible in the logs is this line, that's logged when the browser finally answers with the 504:

  2021/02/08 12:53:09 [error] 24#24: *3046 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.3.10.58, server: sal.local, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8001/", host: "sal.company.tld"

[grahamgilbert]

As I said on slack, I have no experience with your chosen server platform. I would suggest consulting with a rhel expert for possible causes.

[jby]

I know, but still I want an official record of it.