search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
756 stars 85 forks source link

Docker sees a previous image as latest after successfully tagging a new one #925

Open bra-fsn opened 4 years ago

bra-fsn commented 4 years ago

We build docker images on an AWS EC2 instance, pushing them to AWS ECR. The pipeline is controlled by Jenkins. Amongst a lot of other things, the following happens (in this order):

  1. upon triggering, the docker image build is started with the "Docker build and publish" Jenkins plugin (https://plugins.jenkins.io/docker-build-publish/)
  2. the image is tagged (latest) and pushed to AWS ECR
  3. various other jobs are started, including one, which updates Jenkins jobs definitions. These sometimes use the previous docker image as if the latest tag wasn't updated.
  4. this other job prints (for debugging) the result of the docker images command, which -in the failure situation- shows the previous image's hash.

I can't see any errors.

Expected behavior

I would expect that after a successful docker image tagging, all further operations should see the new image as latest.

Actual behavior

Copying the relevant output from the above process: 1.

23:37:55  ---> 566bf6185e42
23:37:55 Successfully built 566bf6185e42

2.

23:37:55 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
23:37:55 [0 Build the om-datasci Docker image] $ docker inspect 566bf6185e42
23:37:55 [0 Build the om-datasci Docker image] $ docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci
[...]
23:38:13 latest: digest: sha256:4da98d187490ed7c0b5e958ea8e7ac467f6494090aaf173ad16e0e6c6ff9cb39 size: 7449

So according to the logs, the image was successfully tagged and uploaded to ECR (verified, however it doesn't really matters here).

The other job reports that it sees the previous image as latest: 4.

+ docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:335604388522451b0726f6039a846ecc9885a0d7e337928166243c71bede3db5   25 hours ago        7.41GB

Steps to reproduce the behavior

Basically what happens here is:

  1. docker build
  2. docker image tag
  3. docker push
  4. docker images print to check which has the latest tag (pretty often the latest, sometimes the previous)

I couldn't yet find a scenario in which I could reproduce this manually.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:29:52 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:28:22 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 105
  Running: 8
  Paused: 0
  Stopped: 97
 Images: 742
 Server Version: 19.03.5
 Storage Driver: aufs
  Root Dir: /var/lib/docker/aufs
  Backing Filesystem: extfs
  Dirs: 1347
  Dirperm1 Supported: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-74-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 61.47GiB
 Name: ip-10-150-14-172
 ID: ZJFE:DJF7:DUGP:WOGK:A66E:IVQS:X6HB:CKP4:SBRG:VROU:ZIBO:GWXF
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: the aufs storage-driver is deprecated, and will be removed in a future release.

Additional environment details (AWS, VirtualBox, physical, etc.)

cpuguy83 commented 4 years ago

Is the job happening on different nodes?

As a side note, I would recommend adding ":latest" on your push command.

bra-fsn commented 4 years ago

Oh, sorry. No, this is all on the same node. That's why I wrote pushing to ecr is irrelevant in this regard, just wanted to make clear we're pushing it somewhere and the image is indeed successfully built.

bra-fsn commented 4 years ago

Removed the Jenkins plugin and converted its work as shell steps and inserted two image listings between a pull after the build, so here's what happens:

16:03:49 + docker build --force-rm=true --build-arg http_proxy=http://172.17.0.2:3128 --build-arg https_proxy=http://172.17.0.2:3128 -t AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest --pull=true --file=model/Dockerfile/om-datasci/Dockerfile .
[...]
16:05:50  ---> aabe1e277667
16:05:50 Successfully built aabe1e277667
16:05:50 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:05:50 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
[...]
16:06:09 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
16:06:10 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:c223c7cd454360ebb5eaa20ca0e1359b8b2a25b67d3808230bf9c29f02de80b2   About an hour ago   7.46GB
16:06:10 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 latest: Pulling from om-datasci
16:06:10 Digest: sha256:d59d36462d5d8279097e80535046b9ba6acedcd7e4e675559c448f5706d07724
16:06:10 Status: Downloaded newer image for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:11 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
16:06:11 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:aabe1e2776673eac4836592bb8bafeb85a08eb97b605d283e63f98a74af43f59   22 seconds ago      7.46GB

Anyone can explain what happens here? BTW, sometimes this works, sometimes it doesn't.

cpuguy83 commented 4 years ago

@tonistiigi Any idea what's happening here?

SHA when doing docker images --no-trunc is different than what is reported by build, and likewise reported after pushing and then pulling.

tonistiigi commented 4 years ago

push/pull digest not matching image sha is expected. One is config other is manifest(compressed). For the diff between the initial build and docker images I have no idea.

bra-fsn commented 4 years ago

Apart from the error described in https://github.com/docker/for-linux/issues/925#issuecomment-586384767 (where the tagging and pushing apparently works, because after a docker pull, the latest tag was the new one) here's another one:

This is the build process' output for 98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12:

15:34:18 Successfully built 98213e2abda0
15:34:18 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:18 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:19 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
15:34:19 6d0d8447a309: Preparing
15:34:19 17362810fca7: Preparing
15:34:19 bafe880a09d7: Preparing
15:34:19 9c5d24974152: Preparing
15:34:19 905722080d38: Preparing
15:34:19 ef2eddffce38: Preparing
15:34:19 3fb063e6172c: Preparing
15:34:19 c78982b74d8f: Preparing
15:34:19 b556bd12de19: Preparing
15:34:19 34ca1dfa0ce4: Preparing
15:34:19 7a980ebdc063: Preparing
15:34:19 97b70daafd93: Preparing
15:34:19 359b43269763: Preparing
15:34:19 b0bfac0637c8: Preparing
15:34:19 fd812d2f0359: Preparing
15:34:19 adc9d64b09c7: Preparing
15:34:19 813c2d431507: Preparing
15:34:19 4a18206c45bb: Preparing
15:34:19 c78982b74d8f: Waiting
15:34:19 d31810a10047: Preparing
15:34:19 3fb063e6172c: Waiting
15:34:19 1a1757b4ffe0: Preparing
15:34:19 5624d338c0cb: Preparing
15:34:19 e201260e6184: Preparing
15:34:19 1133070ab3c8: Preparing
15:34:19 e8f010966380: Preparing
15:34:19 b556bd12de19: Waiting
15:34:19 81c1167eafa0: Preparing
15:34:19 359b43269763: Waiting
15:34:19 7fffe692204a: Preparing
15:34:19 34ca1dfa0ce4: Waiting
15:34:19 04fff313a76c: Preparing
15:34:19 b0bfac0637c8: Waiting
15:34:19 04a55213b645: Preparing
15:34:19 4a18206c45bb: Waiting
15:34:19 7a980ebdc063: Waiting
15:34:19 16542a8fc3be: Preparing
15:34:19 fd812d2f0359: Waiting
15:34:19 adc9d64b09c7: Waiting
15:34:19 97b70daafd93: Waiting
15:34:19 6597da2e2e52: Preparing
15:34:19 813c2d431507: Waiting
15:34:19 977183d4e999: Preparing
15:34:19 e201260e6184: Waiting
15:34:19 d31810a10047: Waiting
15:34:19 1133070ab3c8: Waiting
15:34:19 ef2eddffce38: Waiting
15:34:19 c8be1b8f4d60: Preparing
15:34:19 5624d338c0cb: Waiting
15:34:19 1a1757b4ffe0: Waiting
15:34:19 e8f010966380: Waiting
15:34:19 7fffe692204a: Waiting
15:34:19 16542a8fc3be: Waiting
15:34:19 6597da2e2e52: Waiting
15:34:19 04fff313a76c: Waiting
15:34:19 977183d4e999: Waiting
15:34:19 81c1167eafa0: Waiting
15:34:19 c8be1b8f4d60: Waiting
15:34:19 9c5d24974152: Layer already exists
15:34:19 bafe880a09d7: Layer already exists
15:34:19 905722080d38: Layer already exists
15:34:19 ef2eddffce38: Layer already exists
15:34:19 c78982b74d8f: Layer already exists
15:34:19 3fb063e6172c: Layer already exists
15:34:19 b556bd12de19: Layer already exists
15:34:19 34ca1dfa0ce4: Layer already exists
15:34:19 7a980ebdc063: Layer already exists
15:34:19 97b70daafd93: Layer already exists
15:34:19 359b43269763: Layer already exists
15:34:19 b0bfac0637c8: Layer already exists
15:34:19 fd812d2f0359: Layer already exists
15:34:19 adc9d64b09c7: Layer already exists
15:34:19 813c2d431507: Layer already exists
15:34:19 4a18206c45bb: Layer already exists
15:34:19 d31810a10047: Layer already exists
15:34:19 1a1757b4ffe0: Layer already exists
15:34:19 5624d338c0cb: Layer already exists
15:34:19 1133070ab3c8: Layer already exists
15:34:19 e201260e6184: Layer already exists
15:34:19 e8f010966380: Layer already exists
15:34:19 81c1167eafa0: Layer already exists
15:34:19 7fffe692204a: Layer already exists
15:34:19 04fff313a76c: Layer already exists
15:34:19 04a55213b645: Layer already exists
15:34:19 16542a8fc3be: Layer already exists
15:34:19 6597da2e2e52: Layer already exists
15:34:19 977183d4e999: Layer already exists
15:34:19 c8be1b8f4d60: Layer already exists
15:34:20 6d0d8447a309: Pushed
15:34:36 17362810fca7: Pushed
15:34:37 latest: digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91 size: 7030
15:34:37 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4882421299965629786.sh
15:34:37 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:37 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
15:34:37 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12   19 seconds ago      6.71GB
15:34:37 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins6279250336046699721.sh
15:34:37 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 latest: Pulling from om-datasci
15:34:38 Digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91
15:34:38 Status: Image is up to date for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4701325921360319121.sh
15:34:38 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
15:34:38 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12   20 seconds ago      6.71GB

So the image build succeeded, it was tagged as latest, then pushed to ECR. According to docker images the latest tag was pointing to this image after the build and of course after a docker pull as well. No problems here.

But with the next build:

19:34:15 Successfully built 034a088e4958
19:34:15 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:15 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins5082030717461808757.sh
19:34:15 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:16 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
19:34:16 6d0d8447a309: Preparing
19:34:16 17362810fca7: Preparing
19:34:16 bafe880a09d7: Preparing
19:34:16 9c5d24974152: Preparing
19:34:16 905722080d38: Preparing
19:34:16 ef2eddffce38: Preparing
19:34:16 3fb063e6172c: Preparing
19:34:16 c78982b74d8f: Preparing
19:34:16 b556bd12de19: Preparing
19:34:16 34ca1dfa0ce4: Preparing
19:34:16 7a980ebdc063: Preparing
19:34:16 97b70daafd93: Preparing
19:34:16 359b43269763: Preparing
19:34:16 b0bfac0637c8: Preparing
19:34:16 ef2eddffce38: Waiting
19:34:16 fd812d2f0359: Preparing
19:34:16 adc9d64b09c7: Preparing
19:34:16 3fb063e6172c: Waiting
19:34:16 b556bd12de19: Waiting
19:34:16 813c2d431507: Preparing
19:34:16 c78982b74d8f: Waiting
19:34:16 4a18206c45bb: Preparing
19:34:16 7a980ebdc063: Waiting
19:34:16 d31810a10047: Preparing
19:34:16 b0bfac0637c8: Waiting
19:34:16 97b70daafd93: Waiting
19:34:16 1a1757b4ffe0: Preparing
19:34:16 adc9d64b09c7: Waiting
19:34:16 fd812d2f0359: Waiting
19:34:16 5624d338c0cb: Preparing
19:34:16 359b43269763: Waiting
19:34:16 e201260e6184: Preparing
19:34:16 4a18206c45bb: Waiting
19:34:16 1133070ab3c8: Preparing
19:34:16 d31810a10047: Waiting
19:34:16 813c2d431507: Waiting
19:34:16 e8f010966380: Preparing
19:34:16 1a1757b4ffe0: Waiting
19:34:16 81c1167eafa0: Preparing
19:34:16 e201260e6184: Waiting
19:34:16 5624d338c0cb: Waiting
19:34:16 7fffe692204a: Preparing
19:34:16 04fff313a76c: Preparing
19:34:16 81c1167eafa0: Waiting
19:34:16 7fffe692204a: Waiting
19:34:16 04a55213b645: Preparing
19:34:16 16542a8fc3be: Preparing
19:34:16 6597da2e2e52: Preparing
19:34:16 977183d4e999: Preparing
19:34:16 c8be1b8f4d60: Preparing
19:34:16 977183d4e999: Waiting
19:34:16 04a55213b645: Waiting
19:34:16 16542a8fc3be: Waiting
19:34:16 6597da2e2e52: Waiting
19:34:16 c8be1b8f4d60: Waiting
19:34:16 6d0d8447a309: Layer already exists
19:34:16 905722080d38: Layer already exists
19:34:16 bafe880a09d7: Layer already exists
19:34:16 17362810fca7: Layer already exists
19:34:16 9c5d24974152: Layer already exists
19:34:16 ef2eddffce38: Layer already exists
19:34:16 3fb063e6172c: Layer already exists
19:34:16 c78982b74d8f: Layer already exists
19:34:16 34ca1dfa0ce4: Layer already exists
19:34:16 b556bd12de19: Layer already exists
19:34:16 7a980ebdc063: Layer already exists
19:34:16 97b70daafd93: Layer already exists
19:34:16 b0bfac0637c8: Layer already exists
19:34:16 359b43269763: Layer already exists
19:34:16 fd812d2f0359: Layer already exists
19:34:16 adc9d64b09c7: Layer already exists
19:34:16 813c2d431507: Layer already exists
19:34:16 4a18206c45bb: Layer already exists
19:34:16 d31810a10047: Layer already exists
19:34:16 1a1757b4ffe0: Layer already exists
19:34:16 5624d338c0cb: Layer already exists
19:34:16 e8f010966380: Layer already exists
19:34:16 1133070ab3c8: Layer already exists
19:34:16 e201260e6184: Layer already exists
19:34:16 81c1167eafa0: Layer already exists
19:34:16 04fff313a76c: Layer already exists
19:34:16 7fffe692204a: Layer already exists
19:34:16 04a55213b645: Layer already exists
19:34:16 16542a8fc3be: Layer already exists
19:34:16 6597da2e2e52: Layer already exists
19:34:16 c8be1b8f4d60: Layer already exists
19:34:16 977183d4e999: Layer already exists
19:34:16 latest: digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91 size: 7030
19:34:16 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7257545468652602570.sh
19:34:16 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:16 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
19:34:16 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12   4 hours ago         6.71GB
19:34:16 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins3361005883301152698.sh
19:34:16 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 latest: Pulling from om-datasci
19:34:17 Digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91
19:34:17 Status: Image is up to date for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4617799562635266995.sh
19:34:17 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
19:34:17 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12   4 hours ago         6.71GB

it seems the tagging indeed failed, because the push was referring to the old version.

After the build was restarted by hand, the next one showed:

20:19:33 Successfully built 034a088e4958
20:19:33 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:33 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7550456020519464341.sh
20:19:33 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:34 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
20:19:34 a418db715f1c: Preparing
20:19:34 41c49cc78afd: Preparing
20:19:34 bafe880a09d7: Preparing
20:19:34 9c5d24974152: Preparing
20:19:34 905722080d38: Preparing
20:19:34 ef2eddffce38: Preparing
20:19:34 3fb063e6172c: Preparing
20:19:34 c78982b74d8f: Preparing
20:19:34 b556bd12de19: Preparing
20:19:34 34ca1dfa0ce4: Preparing
20:19:34 7a980ebdc063: Preparing
20:19:34 97b70daafd93: Preparing
20:19:34 359b43269763: Preparing
20:19:34 b0bfac0637c8: Preparing
20:19:34 fd812d2f0359: Preparing
20:19:34 adc9d64b09c7: Preparing
20:19:34 813c2d431507: Preparing
20:19:34 4a18206c45bb: Preparing
20:19:34 d31810a10047: Preparing
20:19:34 1a1757b4ffe0: Preparing
20:19:34 5624d338c0cb: Preparing
20:19:34 ef2eddffce38: Waiting
20:19:34 e201260e6184: Preparing
20:19:34 97b70daafd93: Waiting
20:19:34 3fb063e6172c: Waiting
20:19:34 1133070ab3c8: Preparing
20:19:34 e8f010966380: Preparing
20:19:34 359b43269763: Waiting
20:19:34 81c1167eafa0: Preparing
20:19:34 adc9d64b09c7: Waiting
20:19:34 7fffe692204a: Preparing
20:19:34 b0bfac0637c8: Waiting
20:19:34 fd812d2f0359: Waiting
20:19:34 04fff313a76c: Preparing
20:19:34 813c2d431507: Waiting
20:19:34 04a55213b645: Preparing
20:19:34 b556bd12de19: Waiting
20:19:34 16542a8fc3be: Preparing
20:19:34 7a980ebdc063: Waiting
20:19:34 34ca1dfa0ce4: Waiting
20:19:34 c78982b74d8f: Waiting
20:19:34 6597da2e2e52: Preparing
20:19:34 4a18206c45bb: Waiting
20:19:34 e201260e6184: Waiting
20:19:34 977183d4e999: Preparing
20:19:34 c8be1b8f4d60: Preparing
20:19:34 1133070ab3c8: Waiting
20:19:34 e8f010966380: Waiting
20:19:34 1a1757b4ffe0: Waiting
20:19:34 977183d4e999: Waiting
20:19:34 04fff313a76c: Waiting
20:19:34 7fffe692204a: Waiting
20:19:34 6597da2e2e52: Waiting
20:19:34 16542a8fc3be: Waiting
20:19:34 c8be1b8f4d60: Waiting
20:19:34 5624d338c0cb: Waiting
20:19:34 bafe880a09d7: Layer already exists
20:19:34 905722080d38: Layer already exists
20:19:34 9c5d24974152: Layer already exists
20:19:34 3fb063e6172c: Layer already exists
20:19:34 c78982b74d8f: Layer already exists
20:19:34 ef2eddffce38: Layer already exists
20:19:34 b556bd12de19: Layer already exists
20:19:34 34ca1dfa0ce4: Layer already exists
20:19:34 7a980ebdc063: Layer already exists
20:19:34 97b70daafd93: Layer already exists
20:19:34 359b43269763: Layer already exists
20:19:34 b0bfac0637c8: Layer already exists
20:19:34 fd812d2f0359: Layer already exists
20:19:34 adc9d64b09c7: Layer already exists
20:19:34 813c2d431507: Layer already exists
20:19:34 4a18206c45bb: Layer already exists
20:19:34 1a1757b4ffe0: Layer already exists
20:19:34 d31810a10047: Layer already exists
20:19:34 5624d338c0cb: Layer already exists
20:19:34 e201260e6184: Layer already exists
20:19:34 1133070ab3c8: Layer already exists
20:19:34 e8f010966380: Layer already exists
20:19:34 81c1167eafa0: Layer already exists
20:19:34 7fffe692204a: Layer already exists
20:19:34 04fff313a76c: Layer already exists
20:19:34 04a55213b645: Layer already exists
20:19:34 16542a8fc3be: Layer already exists
20:19:34 6597da2e2e52: Layer already exists
20:19:34 977183d4e999: Layer already exists
20:19:34 c8be1b8f4d60: Layer already exists
20:19:34 a418db715f1c: Pushed
20:19:50 41c49cc78afd: Pushed
20:19:51 latest: digest: sha256:ab8d374ff21528bb19713df2fbd3f8951ba8556237da23aa9e37e8ff4619d88a size: 7030
20:19:51 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins5670137152116218399.sh
20:19:51 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:51 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
20:19:51 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12   5 hours ago         6.71GB
20:19:51 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins8546465052981883902.sh
20:19:51 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 latest: Pulling from om-datasci
20:19:52 Digest: sha256:ab8d374ff21528bb19713df2fbd3f8951ba8556237da23aa9e37e8ff4619d88a
20:19:52 Status: Downloaded newer image for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4842190821063087671.sh
20:19:52 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
20:19:52 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:034a088e49586e9eb1a022f16058f3b209e2be3947c244131aa20aadc329b751   45 minutes ago      6.71GB

In this build:

Any ideas what's going on here?

bra-fsn commented 4 years ago

I've tried with DOCKER_BUILDKIT=1 as well. The effect is the same. A new image is built and is tagged as latest, according to the output:

21:02:40 #9 exporting layers 4.0s done
21:02:40 #9 writing image sha256:fce7a99dd919ea9be1fae77dad9d534eb2bc2e2be4ec6dd1257b83d98aed9ff6 done
21:02:40 #9 naming to AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest done
21:02:40 #9 DONE 4.1s

The push says there are no new layers:

21:02:40 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
21:02:41 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
21:02:41 e024c2664c6b: Preparing
21:02:41 1be4fdd704b0: Preparing
21:02:41 26be1857f4d7: Preparing
21:02:41 06e44bb98520: Preparing
21:02:41 0ada10fce196: Preparing
21:02:41 1a41326226dc: Preparing
21:02:41 e6243216b73a: Preparing
21:02:41 39a79edbb16f: Preparing
21:02:41 f3462ba5b085: Preparing
21:02:41 f9c1597c6dca: Preparing
21:02:41 9b36ebe466d8: Preparing
21:02:41 ce33398259d3: Preparing
21:02:41 7b63f9d2090d: Preparing
21:02:41 ff0839a7f7f4: Preparing
21:02:41 1a41326226dc: Waiting
21:02:41 126d08289624: Preparing
21:02:41 e6243216b73a: Waiting
21:02:41 fd812d2f0359: Preparing
21:02:41 adc9d64b09c7: Preparing
21:02:41 39a79edbb16f: Waiting
21:02:41 813c2d431507: Preparing
21:02:41 9b36ebe466d8: Waiting
21:02:41 4a18206c45bb: Preparing
21:02:41 f3462ba5b085: Waiting
21:02:41 ce33398259d3: Waiting
21:02:41 d31810a10047: Preparing
21:02:41 7b63f9d2090d: Waiting
21:02:41 1a1757b4ffe0: Preparing
21:02:41 f9c1597c6dca: Waiting
21:02:41 fd812d2f0359: Waiting
21:02:41 5624d338c0cb: Preparing
21:02:41 ff0839a7f7f4: Waiting
21:02:41 e201260e6184: Preparing
21:02:41 adc9d64b09c7: Waiting
21:02:41 1133070ab3c8: Preparing
21:02:41 126d08289624: Waiting
21:02:41 4a18206c45bb: Waiting
21:02:41 e8f010966380: Preparing
21:02:41 81c1167eafa0: Preparing
21:02:41 7fffe692204a: Preparing
21:02:41 04fff313a76c: Preparing
21:02:41 04a55213b645: Preparing
21:02:41 5624d338c0cb: Waiting
21:02:41 e201260e6184: Waiting
21:02:41 1133070ab3c8: Waiting
21:02:41 16542a8fc3be: Preparing
21:02:41 6597da2e2e52: Preparing
21:02:41 1a1757b4ffe0: Waiting
21:02:41 e8f010966380: Waiting
21:02:41 977183d4e999: Preparing
21:02:41 81c1167eafa0: Waiting
21:02:41 c8be1b8f4d60: Preparing
21:02:41 04fff313a76c: Waiting
21:02:41 16542a8fc3be: Waiting
21:02:41 977183d4e999: Waiting
21:02:41 6597da2e2e52: Waiting
21:02:41 c8be1b8f4d60: Waiting
21:02:41 06e44bb98520: Layer already exists
21:02:41 0ada10fce196: Layer already exists
21:02:41 26be1857f4d7: Layer already exists
21:02:41 e024c2664c6b: Layer already exists
21:02:41 1be4fdd704b0: Layer already exists
21:02:41 f3462ba5b085: Layer already exists
21:02:41 1a41326226dc: Layer already exists
21:02:41 e6243216b73a: Layer already exists
21:02:41 39a79edbb16f: Layer already exists
21:02:41 f9c1597c6dca: Layer already exists
21:02:41 9b36ebe466d8: Layer already exists
21:02:41 7b63f9d2090d: Layer already exists
21:02:41 ff0839a7f7f4: Layer already exists
21:02:41 126d08289624: Layer already exists
21:02:41 fd812d2f0359: Layer already exists
21:02:41 adc9d64b09c7: Layer already exists
21:02:41 ce33398259d3: Layer already exists
21:02:41 813c2d431507: Layer already exists
21:02:41 1a1757b4ffe0: Layer already exists
21:02:41 d31810a10047: Layer already exists
21:02:41 4a18206c45bb: Layer already exists
21:02:41 5624d338c0cb: Layer already exists
21:02:41 e201260e6184: Layer already exists
21:02:41 1133070ab3c8: Layer already exists
21:02:41 e8f010966380: Layer already exists
21:02:41 04fff313a76c: Layer already exists
21:02:41 7fffe692204a: Layer already exists
21:02:41 81c1167eafa0: Layer already exists
21:02:41 16542a8fc3be: Layer already exists
21:02:41 04a55213b645: Layer already exists
21:02:41 977183d4e999: Layer already exists
21:02:41 c8be1b8f4d60: Layer already exists
21:02:41 6597da2e2e52: Layer already exists
21:02:41 latest: digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f size: 7237

And a docker images command shows latest is still the previous image (built/tagged 38 minutes ago):

21:02:41 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
21:02:41 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
21:02:41 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:337502c14082ce20c670c8a8cd53c4a505a7ee5a1a4c79d6784b35a1923700ee   38 minutes ago      6.75GB
thaJeztah commented 4 years ago

Were there actual changes between the previous (38 minutes ago) and latest build?

bra-fsn commented 4 years ago

Yes, the build process is triggered only on changes in the git repository. Previous run output (albeit it had to do a docker pull to see the new version locally, check the two docker images command output):

20:22:26  > git checkout -f ec6caf01c143c61e559623ac46ba49cd6ab518e5 # timeout=10
[...]
20:23:56 #9 writing image sha256:337502c14082ce20c670c8a8cd53c4a505a7ee5a1a4c79d6784b35a1923700ee done
20:23:56 #9 naming to AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest done
20:23:56 #9 DONE 3.8s
[...]
20:24:15 latest: digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f size: 7237
20:24:15 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins1657966881012180645.sh
20:24:15 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:15 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
20:24:15 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:236ec0b6e4ae6047dc79f19467609c4bbf160f51f0d151cf7599739eb3554078   13 minutes ago      6.75GB
20:24:15 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7728035358095848578.sh
20:24:15 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 latest: Pulling from om-datasci
20:24:16 Digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f
20:24:16 Status: Downloaded newer image for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7542804334142638886.sh
20:24:16 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 REPOSITORY                                                TAG                 IMAGE ID                                                                  CREATED             SIZE
20:24:16 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci   latest              sha256:337502c14082ce20c670c8a8cd53c4a505a7ee5a1a4c79d6784b35a1923700ee   24 seconds ago      6.75GB

and the new (failing) one:

21:02:40 #9 writing image sha256:fce7a99dd919ea9be1fae77dad9d534eb2bc2e2be4ec6dd1257b83d98aed9ff6 done
21:02:40 #9 naming to AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest done
21:02:40 #9 DONE 4.1s
[...]
21:02:41 latest: digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f size: 7237

which has a different hash, but after the push, it writes the old one.