docker-compose build ignoring .dockerignore?

[devinrsmith]

I've got a full example repo here:

https://github.com/devinrsmith/docker-compose-build-test

If I'm using docker-compose build incorrectly, please let me know!

[neciu]

You have problem with file format.

In your .dockerignore change subdir/ to subdir.

[devinrsmith]

Why does the .dockerignore work for docker build -t subdir ../ but not docker-compose build?

[fChristenson]

I am having the same issue running docker-compose 1.2.0. Running vanilla docker build pulls in the ignore file and builds the image, np there. However docker-compose does not use the ignore file, with or without the trailing slash.

[aanand]

Looks like a docker-py issue; a fix is being worked on in https://github.com/docker/docker-py/pull/604. Marking this as a bug to remind us to update our docker-py version once the fix is out.

[thaJeztah]

@aanand looks like https://github.com/docker/docker-py/pull/604 stalled and was removed from the milestone there?

[aanand]

@thaJeztah I've created https://github.com/docker/docker-py/pull/721 to continue the work.

[twillouer]

Hi,

I'm facing one strange problem with .dockerignore:

*.sh

!awesome/script.sh

with docker build all is fine, but docker-compose didn't see "awesome/script.sh"

Am I in the same issue ?

[aanand]

@twillouer that's a known bug, fixed by https://github.com/docker/docker-py/pull/721 - it'll be fixed on the Compose side in 1.5.0.

[twillouer]

thanks !

[ghost]

This still seems to be broken even on latest git master on some machines. It doesn't work for me on a machine given this configuration:

Debian 8.2 with kernel 3.16.0 with lxc-docker version 1.7.1, build 786b29d docker-compose is git master, HEAD is at dabf1e8657674014a5bc89f99edbf2fe0629bb71

The .dockerignore works fine for docker build (which runs instantly), but not for docker-compose (which uploads stuff it shouldn't upload for various minutes).

Also see #2100

[vladkolotvin]

I have same problem. docker-compose ignores ".dockerignore". "docker build" works fine. System: windows 10

[xrstf]

I'm experiencing the same problem using the Docker Toolbox that was released yesterday:

• Windows 7 x64
• Docker version 1.9.0, build 76d6bc9
• Docker-Compose 1.5.0

[dnephin]

It sounds like we're still having problems with .dockerignore. If you're experiencing these problems with compose 1.5.0, please include a sample .dockerignore and directory structure so we can reproduce the error.

[xrstf]

Sorry for not provide a smaller usecase, but this is how I noticed the problem:

I have this package.json for my project:

{
  "dependencies": {
    "grunt-contrib-uglify": "^0.9.2"
  }
}

Plus I have this docker-compose.yml:

web:
  build: .

The Dockerfile looks like this:

FROM node:0.12

My .dockerignore contains one line (I tried adding a trailing slash, but the problem persisted):

node_modules

Now I can do

> npm install
  (...snip...)
> docker build .
Sending build context to Docker daemon  5.12 kB

Awesome, it works, only 5 KiB are sent (node_modules is roughly 10 MiB in total). With docker-compose in Q:\sites\test:

> npm install
  (...snip...)
> docker-compose up
Building web
Traceback (most recent call last):
  File "D:\opt\python\Scripts\docker-compose-script.py", line 9, in <module>
    load_entry_point('docker-compose==1.5.0dev', 'console_scripts', 'docker-compose')()
  File "D:\opt\python\lib\site-packages\compose\cli\main.py", line 54, in main
    command.sys_dispatch()
  File "D:\opt\python\lib\site-packages\compose\cli\docopt_command.py", line 23, in sys_dispatch
    self.dispatch(sys.argv[1:], None)
  File "D:\opt\python\lib\site-packages\compose\cli\docopt_command.py", line 26, in dispatch
    self.perform_command(*self.parse(argv, global_options))
  File "D:\opt\python\lib\site-packages\compose\cli\main.py", line 170, in perform_command
    handler(project, command_options)
  File "D:\opt\python\lib\site-packages\compose\cli\main.py", line 583, in up
    detached=detached
  File "D:\opt\python\lib\site-packages\compose\project.py", line 313, in up
    detached=detached
  File "D:\opt\python\lib\site-packages\compose\service.py", line 404, in execute_convergence_plan
    container = self.create_container(do_build=do_build)
  File "D:\opt\python\lib\site-packages\compose\service.py", line 303, in create_container
    self.ensure_image_exists(do_build=do_build)
  File "D:\opt\python\lib\site-packages\compose\service.py", line 326, in ensure_image_exists
    self.build()
  File "D:\opt\python\lib\site-packages\compose\service.py", line 718, in build
    dockerfile=self.options.get('dockerfile', None),
  File "D:\opt\python\lib\site-packages\docker\api\build.py", line 48, in build
    context = utils.tar(path, exclude=exclude, dockerfile=dockerfile)
  File "D:\opt\python\lib\site-packages\docker\utils\utils.py", line 85, in tar
    t.add(os.path.join(root, path), arcname=path, recursive=False)
  File "D:\opt\python\lib\tarfile.py", line 1998, in add
    tarinfo = self.gettarinfo(name, arcname)
  File "D:\opt\python\lib\tarfile.py", line 1870, in gettarinfo
    statres = os.lstat(name)
WindowsError: [Error 3] Das System kann den angegebenen Pfad nicht finden: 'Q:\\sites\\test\\node_modules\\grunt-contrib-uglify\\node_modules\\maxmin\\node_modules\\pretty-bytes\\node_modules\\meow\\node_modules\\normalize-package-data\\node_modules\\validate-npm-package-license\\node_modules\\spdx-correct\\node_modules\\spdx-license-ids\\spdx-license-ids.json'

(I am using 1.5.0dev on this machine, but I got the exact same problem on my other machine with 1.5.0 final).

With node, I often encounter problems with too long paths and stuff on Windows, but the fact that Docker-Compose is attempting to tar up the node_modules is the indicator that the .dockerignore is ignored.

[ghost]

FWIW, I am still having issues with this on Linux (Ubuntu) with docker-compose 1.5.0dev, so it's probably not isolated to Windows. However, it's on a production machine so I cannot easily put together a minimal test case (it works fine on my testing machine).

[Quantumplation]

I am also having this issue.

Directory structure:

docker-compose.yml
web
  + .dockerignore
  + Dockerfile
  + node_modules
    + ...

docker-compose.yaml

web:
  build: web
  tty: true
  ports:
    - 8081:5000

Dockerfile

FROM microsoft/aspnet

# Curl, node, npm, bower, grunt
RUN apt-get update && apt-get install -y curl
RUN curl -sL https://deb.nodesource.com/setup | bash -
RUN apt-get install -y nodejs
RUN npm install -g bower
RUN npm install -g grunt-bower-cli
RUN npm install -g grunt
RUN npm install -g grunt-cli
RUN npm install -g grunt-bower-task

# Copy the project.json file first, then do a restore.
# This ensures that as long as project.json doesn't change, it will avoid
# doing a package restore
COPY project.json /app/
COPY bower.json /app/
COPY gruntfile.js /app/
COPY package.json /app/
WORKDIR /app

RUN ["dnu", "restore"]

# Then copy the rest of the files
COPY . /app

# Expose the port that the website listens on
EXPOSE 5000
# And start the website
ENTRYPOINT ["dnx", "-p", "project.json", "web"]

.dockerignore

node_modules

Result:

Pi@Ricci MINGW64 /d/proj/Repro
$ docker-compose build
Building web
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "C:\projects\compose\compose\cli\main.py", line 54, in main
  File "C:\projects\compose\compose\cli\docopt_command.py", line 23, in sys_dispatch
  File "C:\projects\compose\compose\cli\docopt_command.py", line 26, in dispatch
  File "C:\projects\compose\compose\cli\main.py", line 171, in perform_command
  File "C:\projects\compose\compose\cli\main.py", line 192, in build
  File "C:\projects\compose\compose\project.py", line 235, in build
  File "C:\projects\compose\compose\service.py", line 683, in build
  File "c:\projects\compose\venv\lib\site-packages\docker\api\build.py", line 48, in build
  File "c:\projects\compose\venv\lib\site-packages\docker\utils\utils.py", line 85, in tar
  File "c:\python27-x64\Lib\tarfile.py", line 2000, in add
  File "c:\python27-x64\Lib\tarfile.py", line 1872, in gettarinfo
WindowsError: [Error 3] The system cannot find the path specified: 'D:\\proj\\Repro\\web\\node_modules\\babel-preset-react\\node_modules\\babel-plugin-transform-react-jsx\\node_modules\\babel-helper-builder-react-jsx\\node_modules\\babel-types\\node_modules\\babel-traverse\\node_modules\\babel-code-frame\\node_modules\\js-tokens\\changelog.md'
docker-compose returned -1

[keatz55]

+1

[keatz55]

Did anybody find a workaround? I am trying to do everything in only the Dockerfile, but haven't had any luck yet.

[esc-rtn]

I'm getting this issue as well, could it be related to the node version everyone is running? Via this issue: https://github.com/npm/npm/issues/3697 .. I imagine upgrading node (more specifically to npm 3+) would fix it, but that's more of a nuclear option, and .dockerignore should still work.

[ghost]

Again despite this being apparently ignored, I am seeing this on a Linux machine as well.

[esc-rtn]

I fixed this issue by removing an unnecessary volume reference in my docker-compose file. Compose ignores .gitignore for volumes.

I also had to upgrade my app to use npm 3+ .. but that may only apply to windows users.

[thaJeztah]

@esc-rtn the .dockerignore file only specifies what files should not be sent to the daemon during build. If you're using a bind-mounted volume during "run", it will simply mount all files in that location as a volume (all files that are present on the host).

[stekershaw]

I'm seeing similar too, on Windows: docker build works as expected, docker-compose not.

I've put a small example on GitHub, with notes in the README.md: https://github.com/stekershaw/docker-compose-ignore

[dnephin]

Another round of fixes for .dockerignore went into docker-py after Compsoe 1.5.2 was released. Could you try out the Compose 1.6.0 RC2 release to see if it's fixed now?

[stekershaw]

Just tried with docker-compose version 1.6.0rc2, build a7636be.

I have three test cases in my repo and with 1.5.2 two of them failed. Now with 1.6.0rc2 only one fails, when I have a directory and a file in the directory excluded, like so:

$ cat .dockerignore
files/test_dir
!files/test_dir/should_be_here_maybe

[aanand]

I get identical behaviour in that case between docker build and docker-compose build.

.dockerignore:

files/test_dir
!files/test_dir/should_be_here_maybe

docker-compose.yml:

test:
  build: .

Dockerfile:

FROM busybox
COPY . /context
CMD ["find", "/context"]

$ docker version
Client:
 Version:      1.10.0-rc1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   677c593
 Built:        Fri Jan 15 18:17:17 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.10.0-rc1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   677c593
 Built:        Fri Jan 15 18:17:17 2016
 OS/Arch:      linux/amd64

$ docker-compose version
docker-compose version 1.6.0rc2, build 695c692
docker-py version: 1.7.0-rc3
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.1j 15 Oct 2014

$ mkdir -p files/test_dir
$ touch files/test_dir/should_be_here_maybe

$ find .
.
./.dockerignore
./docker-compose.yml
./Dockerfile
./files
./files/test_dir
./files/test_dir/should_be_here_maybe

$ docker build --no-cache -t 1607-docker-build .
Sending build context to Docker daemon  5.12 kB
Step 1 : FROM busybox
 ---> 0cb40641836c
Step 2 : COPY . /context
 ---> 859e12600100
Removing intermediate container 9067b263098b
Step 3 : CMD find /context
 ---> Running in 1ddc0a573492
 ---> b1b3beacf5f2
Removing intermediate container 1ddc0a573492
Successfully built b1b3beacf5f2

$ docker run 1607-docker-build
/context
/context/docker-compose.yml
/context/Dockerfile
/context/files
/context/files/test_dir
/context/files/test_dir/should_be_here_maybe
/context/.dockerignore

$ docker-compose build --no-cache
Building test
Step 1 : FROM busybox
 ---> 0cb40641836c
Step 2 : COPY . /context
 ---> d86507051d6d
Removing intermediate container 0af2cbf69b17
Step 3 : CMD find /context
 ---> Running in 8533dae3af74
 ---> 1f736ecb2b38
Removing intermediate container 8533dae3af74
Successfully built 1f736ecb2b38

$ docker-compose run test
/context
/context/docker-compose.yml
/context/Dockerfile
/context/files
/context/files/test_dir
/context/files/test_dir/should_be_here_maybe
/context/.dockerignore

[stekershaw]

Hi @aanand, thanks for you response, I get the same result as you following your steps.

However, the test that I ran previously also had another file present in files/test_dir. If I add another file to files/test_dir then I see that it is not present (as I would expect) with docker build but it is present with docker-compose:

$ touch files/test_dir/should_not_be_here

$ docker build --no-cache -t 1607-docker-build .
Sending build context to Docker daemon  5.12 kB
Step 1 : FROM busybox
 ---> b175bcb79023
Step 2 : COPY . /context
 ---> a23d9645c21c
Removing intermediate container 8eb2bb23c4db
Step 3 : CMD find /context
 ---> Running in d9fef847acd8
 ---> e52ae84b1250
Removing intermediate container d9fef847acd8
Successfully built e52ae84b1250
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

$ docker run --rm 1607-docker-build
/context
/context/Dockerfile
/context/files
/context/files/test_dir
/context/files/test_dir/should_be_here_maybe
/context/docker-compose.yml
/context/.dockerignore

$ docker-compose build --no-cache
Building test
Step 1 : FROM busybox
 ---> b175bcb79023
Step 2 : COPY . /context
 ---> 9df0cf4bfb69
Removing intermediate container 7820f982d59e
Step 3 : CMD find /context
 ---> Running in 06e1a0b89a45
 ---> 2c922dbc66d9
Removing intermediate container 06e1a0b89a45
Successfully built 2c922dbc66d9

$ docker-compose run test
ERROR: Interactive mode is not yet supported on Windows.
Please pass the -d flag when using `docker-compose run`.

$ docker-compose run -d test
dcitest_test_run_2

$ docker logs dcitest_test_run_2
/context
/context/Dockerfile
/context/files
/context/files/test_dir
/context/files/test_dir/should_not_be_here
/context/files/test_dir/should_be_here_maybe
/context/docker-compose.yml
/context/.dockerignore

This is Windows (using git bash as my shell), checked with both with Docker compose 1.5.2 and 1.6.0rc2, docker 1.9.1.

Running the same on Ubuntu 14.04 with docker-compose 1.5.2 and docker 1.9.1 is fine:

# cat .dockerignore
files/test_dir
!files/test_dir/should_be_here_maybe

# find .
.
./.dockerignore
./docker-compose.yml
./files
./files/test_dir
./files/test_dir/should_not_be_here
./files/test_dir/should_be_here_maybe
./Dockerfile

# docker-compose build --no-cache
Building test
Step 1 : FROM busybox
 ---> b175bcb79023
Step 2 : COPY . /context
 ---> c533a0768d5e
Removing intermediate container 0c057fe8eb82
Step 3 : CMD find /context
 ---> Running in e8a0cf1f58d8
 ---> 175777486a25
Removing intermediate container e8a0cf1f58d8
Successfully built 175777486a25

# docker-compose run test
/context
/context/docker-compose.yml
/context/.dockerignore
/context/files
/context/files/test_dir
/context/files/test_dir/should_be_here_maybe
/context/Dockerfile

[bkcummins]

Still dealing with this most of the afternoon.

Docker version 1.10.1, build 9e83765 docker-compose version 1.6.0, build d99cad6

#docker-compose.yml
test:
  build: ./cdn

#Dockerfile
FROM busybox
COPY ["content/","/test/"]
RUN find /test/ -maxdepth 1

#.dockerignore
**/.DS_Store
**/.git
**/.bowerrc
**/bower_components
**/node_modules
**/npm-debug.log

Output is as expected with docker build .

docker-compose build --no-cache test copied over everything

[shin-]

2992 has some more .dockerignore fixes that should be included in 1.6.1

[stekershaw]

Thanks @shin- , unfortunately I still see different behaviour with docker-compose 1.6.2 from Docker Toolbox for Windows:

$ docker-compose --version
docker-compose version 1.6.2, build e80fc83

$ find .
.
./.dockerignore
./docker-compose.yml
./Dockerfile
./files
./files/test_dir
./files/test_dir/should_be_here_maybe
./files/test_dir/should_not_be_here

$ cat .dockerignore
files/test_dir
!files/test_dir/should_be_here_maybe

$ docker-compose build --no-cache
Building test
Step 1 : FROM busybox
latest: Pulling from library/busybox
f810322bba2c: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:97473e34e311e6c1b3f61f2a721d038d1e5eef17d98d1353a513007cf46ca6bd
Status: Downloaded newer image for busybox:latest
 ---> 3240943c9ea3
Step 2 : COPY . /context
 ---> 85a65d7f861c
Removing intermediate container 386d3103d8ab
Step 3 : CMD find /context
 ---> Running in e5e29b5746c4
 ---> 2c2d57a899ea
Removing intermediate container e5e29b5746c4
Successfully built 2c2d57a899ea

$ docker-compose run -d test
dcitest_test_run_1

$ docker logs dcitest_test_run_1
/context
/context/Dockerfile
/context/files
/context/files/test_dir
/context/files/test_dir/should_be_here_maybe
/context/files/test_dir/should_not_be_here
/context/.dockerignore
/context/docker-compose.yml

[shin-]

That's very odd. It definitely works for me.

$ docker-compose build --no-cache
Building web
Step 1 : FROM busybox
 ---> 3240943c9ea3
Step 2 : COPY . /context
 ---> 3619871879ad
Removing intermediate container 08432f688579
Step 3 : CMD find /context
 ---> Running in 5bbcf987c9e7
 ---> cf2bff2c1416
Removing intermediate container 5bbcf987c9e7
Successfully built cf2bff2c1416

$ docker-compose run -d web
Creating network "testdockerignore_default" with the default driver
testdockerignore_web_run_1

$ docker logs testdockerignore_web_run_1 
/context
/context/Dockerfile
/context/files
/context/files/test_dir
/context/files/test_dir/should_be_here
/context/docker-compose.yml
/context/.dockerignore

$ find .
.
./Dockerfile
./files
./files/test_dir
./files/test_dir/should_not_be_here
./files/test_dir/should_be_here
./docker-compose.yml
./.dockerignore

$ cat .dockerignore files/test_dir
!files/test_dir/should_be_here

What's the output of pip show docker-py in your environment?

[digizen]

I believe I also have the same issue.

 Docker build .

works for me. However, enclosing the same build in a docker-compose.yml file breaks my build. In my .dockerignore file, I'm excluding the node_modules directory for my node build. This is the relevant section from my docker-compose.yml:

web:
  build: ./app
  ports:
    - "8080:8080"
  links:
    - mongodb

I believe I'm running the most recent version of docker-compose:

>docker-compose version
docker-compose version 1.6.2, build e80fc83
docker-py version: 1.7.2
CPython version: 2.7.11
OpenSSL version: OpenSSL 1.0.2d 9 Jul 2015

I'm running Windows 10, x64.

[shin-]

WIndows-specific issue, maybe with path separators?

[digizen]

Don't know how to tell whether this is related to path separators.

Let me know if I can assist with testing or otherwise. Right now, I have a batch file that moves the "ignored" folders out of the project directory prior to the build, obviously ugly.

Thanks for your help.

[digizen]

For completeness' sake, this is my .dockerignore file:

**/node_modules

[digizen]

Aha! It's the glob pattern that's causing the problem. removing the "**/" fixed the issue for me.

[digizen]

While I have a workaround for my specific issue, the fact remains that there is a bug. At the very least the glob pattern **/ doesn't get properly parsed with docker-compose, but works fine with the regular docker build command. Other glob patterns may or may not work.

[ghost]

@bfirsh and everyone else who might care:

I have an absolutely simple .dockerignore which looks like this:

livedata
readonly-data

The .dockerignore doesn't work either (those two folders in the build context folder get uploaded to the daemon and it takes forever) and I'm on Linux. I updated just to current 1.6.2, no change. Unless I'm seeing a different problem with the same symptom, let me reiterate again that this doesn't appear to be windows-specific problem..

[nicbarker]

+1, also broken for me on Linux - don't think it's a windows specific problem.

[aanand]

@JonasT @nicbarker How are you confirming that the folders named in .dockerignore are being uploaded, other than the fact it's taking a long time? If you have a way of checking the tarball, I'd like to try and use it to reproduce the problem locally. As it is, I've added some debug to docker-py and am still unable to reproduce:

$ pip list | grep docker
docker-compose (1.7.0.dev0, /Users/aanand/work/docker/compose)
docker-py (1.8.0rc2, /Users/aanand/work/docker/docker-py)

$ find .
.
./.dockerignore
./docker-compose.yml
./Dockerfile
./include.txt
./livedata
./livedata/exclude.txt
./readonly-data
./readonly-data/exclude.txt

$ cat .dockerignore
livedata
readonly-data

$ cat Dockerfile
FROM busybox
COPY . /data

$ docker-compose --verbose build --no-cache
<unrelated output>
docker.utils.utils.tar: Writing tar file to <open file '<fdopen>', mode 'w+b' at 0x1090f26f0>
docker.utils.utils.tar:   Adding .dockerignore
docker.utils.utils.tar:   Adding Dockerfile
docker.utils.utils.tar:   Adding docker-compose.yml
docker.utils.utils.tar:   Adding include.txt
docker.utils.utils.tar: Done
<unrelated output>

[ghost]

@aanand it takes a REALLY long time before it starts, and when I move the Dockerfile to ./context and use "build: ./context/" with no other change it is instantly blazingly fast. Also back then when this problem first appeared I tried docker build manually with "docker build" and it was also instant.

[dnephin]

That doesn't really confirm the issue. If there were another large directory in the project root, moving the build to ./context would make it faster.

[ghost]

@dnephin there are no other folders in there except for "context" itself which I aded as a workaround, and three tiny text files..

[aanand]

@JonasT Is it possible you have a large .git directory that's getting included?

[ghost]

Edit: NVM, it seems I was actually stupid enough to miss a directory because I checked on the wrong rsync'ed copy >.> sorry guys..

I did actually have an issue with this a while back on an older docker-compose version where I investigated it more deeply, but maybe that specific instance I had was actually fixed with some of the recent dockerpy updates.

[digizen]

Back to this being a Windows problem? :)

[ulrichSchreiner]

hi, i have linux, compose 1.6.2, docker 1.10.3 ... exact the same problem. docker uses .dockerignore, compose ignores it.

[aanand]

@ulrichSchreiner Can you provide steps to reproduce?

[ulrichSchreiner]

hi,

here is a gist (https://gist.github.com/ulrichSchreiner/566815cea26ce55b95207e7795cf6962). the .dockerignore contains **/node_modules the Dockerfile adds a file in a t1/a/b/c/node_modules subfolder.

if you build this with "docker build ..." you get an error because there is no file to add (it is correctly ignored because of the pattern in .dockerignore). you can see the output in the last file of the gist.

but if you build it with the given "docker-compose.yml" it is successfully build --> docker-compose ignores the .dockerignore file.

and this is really a pain when your node_modules directory is hundreds of megs ....

i use

docker-compose version 1.6.2, build 4d72027

[aanand]

OK, I can reproduce this. Looks like docker-py's probably got a bug with **/ rules.