nginx 13: Permission denied on AWS ECS

[timaschew]

I changed some config files and locally it works fine. Then I build a new image and uploaded it to AWS using ECS but the webinterface is somehow broken.

Hard Fresh reload looks like this

And like this for a normal reload

The most confusing part is that If I open these missing resources in a new tab, I see the content and it looks good. I think it's because of nginx. Even locally I get sometimes an empty page, but after refreshing it works.

Found errors for nginx lot of these:

2018/01/10 08:01:14 [crit] 17#17: *84 open() "/var/tmp/nginx/proxy/0/02/0000000020" failed (13: Permission denied) while reading upstream, client: 10.223.167.160, server: _, request: "GET /public/build/vendor.2305a8e1d478628b1297.js HTTP/1.1", upstream: "http://127.0.0.1:3000/public/build/vendor.2305a8e1d478628b1297.js", host: "10.126.189.73"

This fits to the ERR_CONTENT_LENGTH_MISMATCH error, but I don't know how to resolve this. This is how the /var/tmp/nginx looks like

bash-4.4# ls -al
total 28
drwx------    7 nginx    nginx         4096 Jan 10 07:36 .
drwxr-xr-x    3 root     root          4096 Dec 21 12:30 ..
drwx------    2 www      root          4096 Jan 10 07:36 client_body
drwx------    2 www      root          4096 Jan 10 07:36 fastcgi
drwx------    2 www      root          4096 Jan 10 07:36 proxy
drwx------    2 www      root          4096 Jan 10 07:36 scgi
drwx------    2 www      root          4096 Jan 10 07:36 uwsgi

BUT this on my local docker container the same, proxy is even there empty and it works.

On AWS I tried to chown everything to www:www and nginx:nginx but still the same problem.

---

BTW: I also changed the Dockerfile a bit, because it was checking out always master which is not really reproducible. Here is the updated Dockerfile: https://gist.github.com/timaschew/3919b8cf79918064ccb0911d29141e22

[ElfoLiNk]

Hello, thank you for reporting. Can you try https://github.com/kamon-io/docker-grafana-graphite/pull/119 ?

[grollinger]

Sadly, that didn't fix the issue.

Same problems for both port 80 and 81.

[Palgie]

I'm also getting this exact issue when deploying over ECS. Any updates on this @timaschew?

[Aethalides]

Did you guys check if selinux is running and maybe you get access violations?

[yohtm]

I was able to fix it by setting the permissions to 777 on /var/tmp/nginx (NOT recursive, just on that folder).

[Aethalides]

I was able to fix it by setting the permissions to 777 on /var/tmp/nginx (NOT recursive, just on that folder).

That's good for testing perhaps but in production I would not recommend that

[vennemp]

@yohtm did you do that on image build or some other mechanism with ECS? currently running into a similar issue

nginx: [emerg] bind() to 0.0.0.0:443 failed (13: Permission denied)

[Aethalides]

@yohtm did you do that on image build or some other mechanism with ECS? currently running into a similar issue

nginx: [emerg] bind() to 0.0.0.0:443 failed (13: Permission denied)

That is NOT the same issue. This error message you are seeing is nginx not being able to bind (to listen) on port 443. Either it is not being started as root, or something else is denying it. like selinx

[vennemp]

Thanks - I am leaning toward selinux or some sort of linux capability like SYS_ADMIN etc. Unfortunately I am using ECS Fargate and Privileged containers are not supported.