nginx serves corrupted files when files edited in host mounted volume

stefanfoulis commented 10 years ago

I get weird file corruption when mounting /var/www into this container. It seems to be specific to nginx in combination with docker-osx (not sure if the same happens with other virtualisation countainers that mount stuff from the host machine into the VM). I could not reproduce this on a pure linux/docker host. I had this on MacOSX 10.9.4 and a fresh docker-osx install.

$ docker-osx shell
$ mkdir ~/Sites/nginxtest
$ docker run --rm -i -t -v /Users/stefanfoulis/Sites/nginxtest:/usr/local/nginx/html -p 0.0.0.0:42002:80 nginx:latest
$ curl http://172.16.42.43:42002/index.html
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.7.1</center>
</body>
</html>

All fine so far.

$ echo "Hi there" > ~/Sites/nginxtest/index.html
$ curl http://172.16.42.43:42002/index.html
Hi there

So far so good. Let's add something to the file

$ echo "More stuff" >> ~/Sites/nginxtest/index.html
$ curl http://172.16.42.43:42002/
Hi there

The new line is missing. Even stopping the container and starting a new one will not have the change show up.

^C%
$ docker run --rm -i -t -v /Users/stefanfoulis/Sites/nginxtest:/usr/local/nginx/html -p 0.0.0.0:42002:80 nginx:latest
$ curl http://172.16.42.43:42002/
Hi there

And what is super weird, I can connect to the container with bash, run nginx in the backround and observe the files changing in the filesystem. But nginx does not pick them up.

$ docker run --rm -i -t -v /Users/stefanfoulis/Sites/nginxtest:/usr/local/nginx/html -p 0.0.0.0:42002:80 nginx:latest /bin/bash
root@e9e8fae26a58:/usr/local/nginx/html# nginx &
[1] 8
root@e9e8fae26a58:/usr/local/nginx/html# cat index.html
Hi there
More stuff

but at the same time from the host:

$ curl http://172.16.42.43:42002/index.html
Hi there

Deleting the file gets picked up though

$ rm ~/Sites/nginxtest/index.html
$ curl http://172.16.42.43:42002/index.html
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.7.1</center>
</body>
</html>

And now it gets interesting. Overwriting a file with a shorter content will show the content of the old file, but truncated at the length of the new file.

$ echo "And now for something completely different." > ~/Sites/nginxtest/index.html
$ curl http://172.16.42.43:42002/index.html
And now for something completely different.
$ echo "Now a shorter text" > ~/Sites/nginxtest/index.html
$ curl http://172.16.42.43:42002/index.html
And now for somethi%
$ echo "A way longer text that contains completely different content. Did I mention it is longer?" > ~/Sites/nginxtest/index.html
$ curl http://172.16.42.43:42002/index.html                                                                                        
And now for something completely different.

It looks like it's tied to the first request done with a new file. If you delete the file and make at least one request for that file and get a 404 and then re-add it, the new file contents will be picked up.

It seems to be a combination of docker-osx and nginx (I experienced this with the orchardup/nginx docker image as well. We did a quick test with an apache image, which did not show this behaviour.

My wild speculation: I suspect nginx is doing some deep magic in the way it detects file changes and caches it somehow (but apparently not just in memory, otherwise it would not survive container destruction and restart). Something with the docker abstraction of the volumes (and maybe just in combination of using a volume that was shared to the vm from the host os) and how nginx reads the files.

I'm linking this issue from docker and the official nginx docker image as well, as I'm not sure where the issue originates.

Some additional environment information:

docker version
Client version: 1.1.1
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): bd609d2
Server version: 1.1.1
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): bd609d2

ghost commented 10 years ago

I am having the exact same issue. Same versions.. same behaviour.

I think it is related to the VirtualBox sendfile issue.

What solved it for me was adding sendfile off; to the location tag in my site configuration. Which is strange because I had it already disabled in my main nginx config (which used to work in the past).

amosyuen commented 9 years ago

same problem, mrmaxon's workaround works for me

joserobleda commented 9 years ago

same problem here, and it works with sendfile off;

cor commented 8 years ago

I'm having the same issue, this really needs to be fixed

virtustilus commented 7 years ago

I had to restart docker-machine and after that all became ok: docker-machine stop default

julien-duponchelle / docker-osx

nginx serves corrupted files when files edited in host mounted volume #68