Problem with nginx container. https://localhost:3000 is not working.

[Pr0fe5s0r]

I have installed all the containers and started them. Everything is Working great! But the Nginx container is not working correctly. But the container shows nginx server is running in port 3000:80.

This is the log.

2023-07-27 22:32:43 2023/07/27 17:02:43 [notice] 1#1: signal 29 (SIGIO) received 2023-07-27 22:32:43 2023/07/27 17:02:43 [notice] 1#1: signal 17 (SIGCHLD) received from 16 2023-07-27 22:32:43 2023/07/27 17:02:43 [notice] 1#1: worker process 16 exited with code 0 2023-07-27 22:32:43 2023/07/27 17:02:43 [notice] 1#1: exit 2023-07-27 22:32:44 /bin/sh: can't open /etc/nginx/conf.d/app.conf.template.dev: no such file 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: using the "epoll" event method 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: nginx/1.23.4 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: built by gcc 12.2.1 20220924 (Alpine 12.2.1_git20220924-r4) 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: OS: Linux 5.15.90.1-microsoft-standard-WSL2 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker processes 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 8 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 9 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 10 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 11 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 12 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 13 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 14 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 15 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 16 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 17 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 18 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 19 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 20 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 21 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 22 2023-07-27 22:32:44 2023/07/27 17:02:44 [notice] 1#1: start worker process 23

i think this is the problem 2023-07-27 22:32:44 /bin/sh: can't open /etc/nginx/conf.d/app.conf.template.dev: no such file

It could be great if anyone knows the solution.

[derekwelton]

I'm having the exact same issue.

[derekwelton]

I left it alone and come to find out after 20-30min, it started working. Is there a reason it takes this long?

[pkabra]

This line from your log:

2023-07-27 22:32:44 /bin/sh: can't open /etc/nginx/conf.d/app.conf.template.dev: no such file

indicates that the host folder deployment/data/nginx mounted at /etc/nginx/conf.d was somehow missing the app.conf.template.dev file. The command in docker-compose.dev.yaml for nginx uses that file to initialize the Nginx app config, which configures the reverse proxy for the web app and API server.

I've run into this occasionally as well, and I'm surprised more people haven't. In my case, I think what is happening is that there's some latency when mounting folders inside the container, and the volume mount is still syncing files from the host machine to the container when the run command gets executed, so the files appear missing when nginx launches. The solution I was using was to just restart the container a whole bunch of times until it finally launched with the app.conf file correctly created and mounted in the folder when Nginx started running. I've seen some posts complaining that in some cases the filesystem latency can be as much as 600ms+ when using gRPC FUSE. This is just a theory though.

My setup is an M1 Mac with macOS Ventura 13.4 running Docker Desktop 4.21.1

Based on this theory here's what I'd recommend trying:

1. If you're on Docker Desktop on a Mac running macOS Ventura or higher (13 or higher), go into Docker Desktop settings, and under "General" check the box to "Use Virtualization framework". That might improve performance slightly. You can also try switching to "VirtioFS" for the file sharing implementation right under the virtualization framework checkbox, and see if that also helps.
2. If that doesn't solve things OR you're not running a Mac or Docker desktop, try adding a sleep to the beginning of the command under the nginx container in docker-compose.dev.yaml. For example, the new command could be: /bin/sh -c "sleep 2 && envsubst '$$\{DOMAIN\}' < /etc/nginx/conf.d/app.conf.template.dev > /etc/nginx/conf.d/app.conf && while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"" So the container starts off waiting for 2 seconds to let the volumes sync properly before kicking off the remaining commands.

If despite all this you're still running into problems, then there's probably something else going on, and we'd have to dig deeper.

Personally, I've not run into the problem since switching to VirtioFS with Virtualization Framework turned on, but your mileage may vary.

Hope that helps.

[yuhongsun96]

Any luck getting through this issue? Happy to jump in the conversation and provide help

[yuhongsun96]

After hearing about this issue many times, it seems nginx can occasionally get in a bad state if the other services are too slow in coming up. This can be trivially fixed by restarting the nginx container.

We will get around to fixing this, it's not super high priority as the fix is very simple, but we will definitely get around to it!

[clutchwave]

This can be trivially fixed by restarting the nginx container.

This just happened to me, tinkering around with danswer. sudo docker container restart danswer-stack-nginx-1 fixed it. Thanks!