search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
756 stars 85 forks source link

docker build race condition - intermediate container dead or removed before commit #574

Open xf57212 opened 5 years ago

xf57212 commented 5 years ago

Expected behavior

docker build -t <image name>:<version> <Dockerfile location> to build without issue

Actual behavior

Occasionally (4 in 31 builds), docker build -t <image name>:<version> <Dockerfile location> will fail to build with the error cannot commit container <container id> which is dead or cannot commit container <container id> which is being removed

Steps to reproduce the behavior

Use docker build -t <image name>:<version> and sometimes it will happen.

Our Dockerfiles start with a centos7 image as a base, but this is in a private repo.

Sample Dockerfile:

FROM REDACTED

ARG env

ENV PROPS_PATH=/etc/REDACTED/REDACTED

COPY src/main/repo/* /etc/yum.repos.d/

COPY src/main/bin/ /var/lib/REDACTED/REDACTED/
RUN chmod 0755 /var/lib/REDACTED/REDACTED/tool/*

COPY src/main/etc/conf/ /etc/conf/
RUN chmod 0644 /etc/conf/*

RUN yum install -y java-1.8.0-openjdk.x86_64

RUN mkdir -p /data/tmp
RUN chmod ug+rwx /data/tmp

RUN mkdir -p $PROPS_PATH
COPY src/main/etc/conf/ $PROPS_PATH
COPY build/libs/*.jar /var/lib/REDACTED/REDACTED/service.jar

RUN sh -c 'touch /var/lib/REDACTED/REDACTED/service.jar'
RUN chmod a+x /var/lib/REDACTED/REDACTED/service.jar

CMD "cp REDACTED REDACTED"
CMD "chmod 0644 REDACTED"

ENV JAVA_OPTS="REDACTED"
ENTRYPOINT java $JAVA_OPTS -jar /var/lib/REDACTED/REDACTED/service.jar REDACTED

Output of docker version:

Client:
 Version:       17.12.0-ce
 API version:   1.35
 Go version:    go1.9.2
 Git commit:    c97c6d6
 Built: Wed Dec 27 20:10:14 2017
 OS/Arch:       linux/amd64

Server:
 Engine:
  Version:      17.12.0-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2
  Git commit:   c97c6d6
  Built:        c97c6d6
  OS/Arch:      Wed Dec 27 20:12:46 2016
  Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 17.12.0-ce
Storage Driver: devicemapper
 Pool NameL docker-253:5-4235488-pool
 Pool Blocksize: 65.54kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Udev Sync Supported: true
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metatadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 2.476GB
 Data Space Total: 107.4GB
 Data Space Available: 1.851GB
 Metadata Space Used: 3.879MB
 Metadata Space Total: 2.147GB
 Metadata Space Available: 1.851GB
 Thin Pool Minimum Free Space: 0.74GB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: 9c461c8187koph8rhurai5jb
 Is Manager: true
 ClusterID: kpf8t76ud3gtsbwyuqz8b6hsx
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 0
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Hearbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: 10.0.124.54
 Manager Addresses:
  10.0.124.54.2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.11.6.e17.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.638GiB
Name: REDACTED
ID: REDACTED
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: devicemapper: usage of loopbaack devices is strongly discouraged for production use.
         Use '--storage-opt dm.thinpooldev' to specify a custom block storage device.

Additional environment details (AWS, VirtualBox, physical, etc.)

xf57212 commented 5 years ago

If a build fails with one of these error messages, simply rerunning the build without changing anything usually works. However the builds generally take 10 minutes a piece so it is a decently large waste of time if a handful need to be rerun.