Open bra-fsn opened 4 years ago
Is the job happening on different nodes?
As a side note, I would recommend adding ":latest" on your push command.
Oh, sorry. No, this is all on the same node. That's why I wrote pushing to ecr is irrelevant in this regard, just wanted to make clear we're pushing it somewhere and the image is indeed successfully built.
Removed the Jenkins plugin and converted its work as shell steps and inserted two image listings between a pull after the build, so here's what happens:
16:03:49 + docker build --force-rm=true --build-arg http_proxy=http://172.17.0.2:3128 --build-arg https_proxy=http://172.17.0.2:3128 -t AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest --pull=true --file=model/Dockerfile/om-datasci/Dockerfile .
[...]
16:05:50 ---> aabe1e277667
16:05:50 Successfully built aabe1e277667
16:05:50 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:05:50 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
[...]
16:06:09 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 REPOSITORY TAG IMAGE ID CREATED SIZE
16:06:10 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:c223c7cd454360ebb5eaa20ca0e1359b8b2a25b67d3808230bf9c29f02de80b2 About an hour ago 7.46GB
16:06:10 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 latest: Pulling from om-datasci
16:06:10 Digest: sha256:d59d36462d5d8279097e80535046b9ba6acedcd7e4e675559c448f5706d07724
16:06:10 Status: Downloaded newer image for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:10 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
16:06:11 REPOSITORY TAG IMAGE ID CREATED SIZE
16:06:11 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:aabe1e2776673eac4836592bb8bafeb85a08eb97b605d283e63f98a74af43f59 22 seconds ago 7.46GB
Anyone can explain what happens here? BTW, sometimes this works, sometimes it doesn't.
@tonistiigi Any idea what's happening here?
SHA when doing docker images --no-trunc
is different than what is reported by build, and likewise reported after pushing and then pulling.
push/pull digest not matching image sha is expected. One is config other is manifest(compressed). For the diff between the initial build and docker images
I have no idea.
Apart from the error described in https://github.com/docker/for-linux/issues/925#issuecomment-586384767 (where the tagging and pushing apparently works, because after a docker pull
, the latest
tag was the new one) here's another one:
This is the build process' output for 98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12
:
15:34:18 Successfully built 98213e2abda0
15:34:18 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:18 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:19 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
15:34:19 6d0d8447a309: Preparing
15:34:19 17362810fca7: Preparing
15:34:19 bafe880a09d7: Preparing
15:34:19 9c5d24974152: Preparing
15:34:19 905722080d38: Preparing
15:34:19 ef2eddffce38: Preparing
15:34:19 3fb063e6172c: Preparing
15:34:19 c78982b74d8f: Preparing
15:34:19 b556bd12de19: Preparing
15:34:19 34ca1dfa0ce4: Preparing
15:34:19 7a980ebdc063: Preparing
15:34:19 97b70daafd93: Preparing
15:34:19 359b43269763: Preparing
15:34:19 b0bfac0637c8: Preparing
15:34:19 fd812d2f0359: Preparing
15:34:19 adc9d64b09c7: Preparing
15:34:19 813c2d431507: Preparing
15:34:19 4a18206c45bb: Preparing
15:34:19 c78982b74d8f: Waiting
15:34:19 d31810a10047: Preparing
15:34:19 3fb063e6172c: Waiting
15:34:19 1a1757b4ffe0: Preparing
15:34:19 5624d338c0cb: Preparing
15:34:19 e201260e6184: Preparing
15:34:19 1133070ab3c8: Preparing
15:34:19 e8f010966380: Preparing
15:34:19 b556bd12de19: Waiting
15:34:19 81c1167eafa0: Preparing
15:34:19 359b43269763: Waiting
15:34:19 7fffe692204a: Preparing
15:34:19 34ca1dfa0ce4: Waiting
15:34:19 04fff313a76c: Preparing
15:34:19 b0bfac0637c8: Waiting
15:34:19 04a55213b645: Preparing
15:34:19 4a18206c45bb: Waiting
15:34:19 7a980ebdc063: Waiting
15:34:19 16542a8fc3be: Preparing
15:34:19 fd812d2f0359: Waiting
15:34:19 adc9d64b09c7: Waiting
15:34:19 97b70daafd93: Waiting
15:34:19 6597da2e2e52: Preparing
15:34:19 813c2d431507: Waiting
15:34:19 977183d4e999: Preparing
15:34:19 e201260e6184: Waiting
15:34:19 d31810a10047: Waiting
15:34:19 1133070ab3c8: Waiting
15:34:19 ef2eddffce38: Waiting
15:34:19 c8be1b8f4d60: Preparing
15:34:19 5624d338c0cb: Waiting
15:34:19 1a1757b4ffe0: Waiting
15:34:19 e8f010966380: Waiting
15:34:19 7fffe692204a: Waiting
15:34:19 16542a8fc3be: Waiting
15:34:19 6597da2e2e52: Waiting
15:34:19 04fff313a76c: Waiting
15:34:19 977183d4e999: Waiting
15:34:19 81c1167eafa0: Waiting
15:34:19 c8be1b8f4d60: Waiting
15:34:19 9c5d24974152: Layer already exists
15:34:19 bafe880a09d7: Layer already exists
15:34:19 905722080d38: Layer already exists
15:34:19 ef2eddffce38: Layer already exists
15:34:19 c78982b74d8f: Layer already exists
15:34:19 3fb063e6172c: Layer already exists
15:34:19 b556bd12de19: Layer already exists
15:34:19 34ca1dfa0ce4: Layer already exists
15:34:19 7a980ebdc063: Layer already exists
15:34:19 97b70daafd93: Layer already exists
15:34:19 359b43269763: Layer already exists
15:34:19 b0bfac0637c8: Layer already exists
15:34:19 fd812d2f0359: Layer already exists
15:34:19 adc9d64b09c7: Layer already exists
15:34:19 813c2d431507: Layer already exists
15:34:19 4a18206c45bb: Layer already exists
15:34:19 d31810a10047: Layer already exists
15:34:19 1a1757b4ffe0: Layer already exists
15:34:19 5624d338c0cb: Layer already exists
15:34:19 1133070ab3c8: Layer already exists
15:34:19 e201260e6184: Layer already exists
15:34:19 e8f010966380: Layer already exists
15:34:19 81c1167eafa0: Layer already exists
15:34:19 7fffe692204a: Layer already exists
15:34:19 04fff313a76c: Layer already exists
15:34:19 04a55213b645: Layer already exists
15:34:19 16542a8fc3be: Layer already exists
15:34:19 6597da2e2e52: Layer already exists
15:34:19 977183d4e999: Layer already exists
15:34:19 c8be1b8f4d60: Layer already exists
15:34:20 6d0d8447a309: Pushed
15:34:36 17362810fca7: Pushed
15:34:37 latest: digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91 size: 7030
15:34:37 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4882421299965629786.sh
15:34:37 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:37 REPOSITORY TAG IMAGE ID CREATED SIZE
15:34:37 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12 19 seconds ago 6.71GB
15:34:37 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins6279250336046699721.sh
15:34:37 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 latest: Pulling from om-datasci
15:34:38 Digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91
15:34:38 Status: Image is up to date for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4701325921360319121.sh
15:34:38 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
15:34:38 REPOSITORY TAG IMAGE ID CREATED SIZE
15:34:38 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12 20 seconds ago 6.71GB
So the image build succeeded, it was tagged as latest, then pushed to ECR. According to docker images
the latest tag was pointing to this image after the build and of course after a docker pull
as well.
No problems here.
But with the next build:
19:34:15 Successfully built 034a088e4958
19:34:15 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:15 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins5082030717461808757.sh
19:34:15 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:16 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
19:34:16 6d0d8447a309: Preparing
19:34:16 17362810fca7: Preparing
19:34:16 bafe880a09d7: Preparing
19:34:16 9c5d24974152: Preparing
19:34:16 905722080d38: Preparing
19:34:16 ef2eddffce38: Preparing
19:34:16 3fb063e6172c: Preparing
19:34:16 c78982b74d8f: Preparing
19:34:16 b556bd12de19: Preparing
19:34:16 34ca1dfa0ce4: Preparing
19:34:16 7a980ebdc063: Preparing
19:34:16 97b70daafd93: Preparing
19:34:16 359b43269763: Preparing
19:34:16 b0bfac0637c8: Preparing
19:34:16 ef2eddffce38: Waiting
19:34:16 fd812d2f0359: Preparing
19:34:16 adc9d64b09c7: Preparing
19:34:16 3fb063e6172c: Waiting
19:34:16 b556bd12de19: Waiting
19:34:16 813c2d431507: Preparing
19:34:16 c78982b74d8f: Waiting
19:34:16 4a18206c45bb: Preparing
19:34:16 7a980ebdc063: Waiting
19:34:16 d31810a10047: Preparing
19:34:16 b0bfac0637c8: Waiting
19:34:16 97b70daafd93: Waiting
19:34:16 1a1757b4ffe0: Preparing
19:34:16 adc9d64b09c7: Waiting
19:34:16 fd812d2f0359: Waiting
19:34:16 5624d338c0cb: Preparing
19:34:16 359b43269763: Waiting
19:34:16 e201260e6184: Preparing
19:34:16 4a18206c45bb: Waiting
19:34:16 1133070ab3c8: Preparing
19:34:16 d31810a10047: Waiting
19:34:16 813c2d431507: Waiting
19:34:16 e8f010966380: Preparing
19:34:16 1a1757b4ffe0: Waiting
19:34:16 81c1167eafa0: Preparing
19:34:16 e201260e6184: Waiting
19:34:16 5624d338c0cb: Waiting
19:34:16 7fffe692204a: Preparing
19:34:16 04fff313a76c: Preparing
19:34:16 81c1167eafa0: Waiting
19:34:16 7fffe692204a: Waiting
19:34:16 04a55213b645: Preparing
19:34:16 16542a8fc3be: Preparing
19:34:16 6597da2e2e52: Preparing
19:34:16 977183d4e999: Preparing
19:34:16 c8be1b8f4d60: Preparing
19:34:16 977183d4e999: Waiting
19:34:16 04a55213b645: Waiting
19:34:16 16542a8fc3be: Waiting
19:34:16 6597da2e2e52: Waiting
19:34:16 c8be1b8f4d60: Waiting
19:34:16 6d0d8447a309: Layer already exists
19:34:16 905722080d38: Layer already exists
19:34:16 bafe880a09d7: Layer already exists
19:34:16 17362810fca7: Layer already exists
19:34:16 9c5d24974152: Layer already exists
19:34:16 ef2eddffce38: Layer already exists
19:34:16 3fb063e6172c: Layer already exists
19:34:16 c78982b74d8f: Layer already exists
19:34:16 34ca1dfa0ce4: Layer already exists
19:34:16 b556bd12de19: Layer already exists
19:34:16 7a980ebdc063: Layer already exists
19:34:16 97b70daafd93: Layer already exists
19:34:16 b0bfac0637c8: Layer already exists
19:34:16 359b43269763: Layer already exists
19:34:16 fd812d2f0359: Layer already exists
19:34:16 adc9d64b09c7: Layer already exists
19:34:16 813c2d431507: Layer already exists
19:34:16 4a18206c45bb: Layer already exists
19:34:16 d31810a10047: Layer already exists
19:34:16 1a1757b4ffe0: Layer already exists
19:34:16 5624d338c0cb: Layer already exists
19:34:16 e8f010966380: Layer already exists
19:34:16 1133070ab3c8: Layer already exists
19:34:16 e201260e6184: Layer already exists
19:34:16 81c1167eafa0: Layer already exists
19:34:16 04fff313a76c: Layer already exists
19:34:16 7fffe692204a: Layer already exists
19:34:16 04a55213b645: Layer already exists
19:34:16 16542a8fc3be: Layer already exists
19:34:16 6597da2e2e52: Layer already exists
19:34:16 c8be1b8f4d60: Layer already exists
19:34:16 977183d4e999: Layer already exists
19:34:16 latest: digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91 size: 7030
19:34:16 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7257545468652602570.sh
19:34:16 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:16 REPOSITORY TAG IMAGE ID CREATED SIZE
19:34:16 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12 4 hours ago 6.71GB
19:34:16 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins3361005883301152698.sh
19:34:16 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 latest: Pulling from om-datasci
19:34:17 Digest: sha256:1103e635a12d46cf85df6597443f59ad55363c8b0caa30b63697a9af6cee3d91
19:34:17 Status: Image is up to date for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4617799562635266995.sh
19:34:17 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
19:34:17 REPOSITORY TAG IMAGE ID CREATED SIZE
19:34:17 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12 4 hours ago 6.71GB
it seems the tagging indeed failed, because the push was referring to the old version.
After the build was restarted by hand, the next one showed:
20:19:33 Successfully built 034a088e4958
20:19:33 Successfully tagged AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:33 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7550456020519464341.sh
20:19:33 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:34 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
20:19:34 a418db715f1c: Preparing
20:19:34 41c49cc78afd: Preparing
20:19:34 bafe880a09d7: Preparing
20:19:34 9c5d24974152: Preparing
20:19:34 905722080d38: Preparing
20:19:34 ef2eddffce38: Preparing
20:19:34 3fb063e6172c: Preparing
20:19:34 c78982b74d8f: Preparing
20:19:34 b556bd12de19: Preparing
20:19:34 34ca1dfa0ce4: Preparing
20:19:34 7a980ebdc063: Preparing
20:19:34 97b70daafd93: Preparing
20:19:34 359b43269763: Preparing
20:19:34 b0bfac0637c8: Preparing
20:19:34 fd812d2f0359: Preparing
20:19:34 adc9d64b09c7: Preparing
20:19:34 813c2d431507: Preparing
20:19:34 4a18206c45bb: Preparing
20:19:34 d31810a10047: Preparing
20:19:34 1a1757b4ffe0: Preparing
20:19:34 5624d338c0cb: Preparing
20:19:34 ef2eddffce38: Waiting
20:19:34 e201260e6184: Preparing
20:19:34 97b70daafd93: Waiting
20:19:34 3fb063e6172c: Waiting
20:19:34 1133070ab3c8: Preparing
20:19:34 e8f010966380: Preparing
20:19:34 359b43269763: Waiting
20:19:34 81c1167eafa0: Preparing
20:19:34 adc9d64b09c7: Waiting
20:19:34 7fffe692204a: Preparing
20:19:34 b0bfac0637c8: Waiting
20:19:34 fd812d2f0359: Waiting
20:19:34 04fff313a76c: Preparing
20:19:34 813c2d431507: Waiting
20:19:34 04a55213b645: Preparing
20:19:34 b556bd12de19: Waiting
20:19:34 16542a8fc3be: Preparing
20:19:34 7a980ebdc063: Waiting
20:19:34 34ca1dfa0ce4: Waiting
20:19:34 c78982b74d8f: Waiting
20:19:34 6597da2e2e52: Preparing
20:19:34 4a18206c45bb: Waiting
20:19:34 e201260e6184: Waiting
20:19:34 977183d4e999: Preparing
20:19:34 c8be1b8f4d60: Preparing
20:19:34 1133070ab3c8: Waiting
20:19:34 e8f010966380: Waiting
20:19:34 1a1757b4ffe0: Waiting
20:19:34 977183d4e999: Waiting
20:19:34 04fff313a76c: Waiting
20:19:34 7fffe692204a: Waiting
20:19:34 6597da2e2e52: Waiting
20:19:34 16542a8fc3be: Waiting
20:19:34 c8be1b8f4d60: Waiting
20:19:34 5624d338c0cb: Waiting
20:19:34 bafe880a09d7: Layer already exists
20:19:34 905722080d38: Layer already exists
20:19:34 9c5d24974152: Layer already exists
20:19:34 3fb063e6172c: Layer already exists
20:19:34 c78982b74d8f: Layer already exists
20:19:34 ef2eddffce38: Layer already exists
20:19:34 b556bd12de19: Layer already exists
20:19:34 34ca1dfa0ce4: Layer already exists
20:19:34 7a980ebdc063: Layer already exists
20:19:34 97b70daafd93: Layer already exists
20:19:34 359b43269763: Layer already exists
20:19:34 b0bfac0637c8: Layer already exists
20:19:34 fd812d2f0359: Layer already exists
20:19:34 adc9d64b09c7: Layer already exists
20:19:34 813c2d431507: Layer already exists
20:19:34 4a18206c45bb: Layer already exists
20:19:34 1a1757b4ffe0: Layer already exists
20:19:34 d31810a10047: Layer already exists
20:19:34 5624d338c0cb: Layer already exists
20:19:34 e201260e6184: Layer already exists
20:19:34 1133070ab3c8: Layer already exists
20:19:34 e8f010966380: Layer already exists
20:19:34 81c1167eafa0: Layer already exists
20:19:34 7fffe692204a: Layer already exists
20:19:34 04fff313a76c: Layer already exists
20:19:34 04a55213b645: Layer already exists
20:19:34 16542a8fc3be: Layer already exists
20:19:34 6597da2e2e52: Layer already exists
20:19:34 977183d4e999: Layer already exists
20:19:34 c8be1b8f4d60: Layer already exists
20:19:34 a418db715f1c: Pushed
20:19:50 41c49cc78afd: Pushed
20:19:51 latest: digest: sha256:ab8d374ff21528bb19713df2fbd3f8951ba8556237da23aa9e37e8ff4619d88a size: 7030
20:19:51 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins5670137152116218399.sh
20:19:51 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:51 REPOSITORY TAG IMAGE ID CREATED SIZE
20:19:51 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:98213e2abda0bbf7decf2ea7d0454a5bd9fadd7c790e8563120da00cf6f16c12 5 hours ago 6.71GB
20:19:51 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins8546465052981883902.sh
20:19:51 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 latest: Pulling from om-datasci
20:19:52 Digest: sha256:ab8d374ff21528bb19713df2fbd3f8951ba8556237da23aa9e37e8ff4619d88a
20:19:52 Status: Downloaded newer image for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins4842190821063087671.sh
20:19:52 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:19:52 REPOSITORY TAG IMAGE ID CREATED SIZE
20:19:52 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:034a088e49586e9eb1a022f16058f3b209e2be3947c244131aa20aadc329b751 45 minutes ago 6.71GB
In this build:
034a088e4958
) and actually happeneddocker images
command showed the previous version as latest
docker pull
Any ideas what's going on here?
I've tried with DOCKER_BUILDKIT=1
as well. The effect is the same.
A new image is built and is tagged as latest, according to the output:
21:02:40 #9 exporting layers 4.0s done
21:02:40 #9 writing image sha256:fce7a99dd919ea9be1fae77dad9d534eb2bc2e2be4ec6dd1257b83d98aed9ff6 done
21:02:40 #9 naming to AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest done
21:02:40 #9 DONE 4.1s
The push says there are no new layers:
21:02:40 + docker push AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
21:02:41 The push refers to repository [AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci]
21:02:41 e024c2664c6b: Preparing
21:02:41 1be4fdd704b0: Preparing
21:02:41 26be1857f4d7: Preparing
21:02:41 06e44bb98520: Preparing
21:02:41 0ada10fce196: Preparing
21:02:41 1a41326226dc: Preparing
21:02:41 e6243216b73a: Preparing
21:02:41 39a79edbb16f: Preparing
21:02:41 f3462ba5b085: Preparing
21:02:41 f9c1597c6dca: Preparing
21:02:41 9b36ebe466d8: Preparing
21:02:41 ce33398259d3: Preparing
21:02:41 7b63f9d2090d: Preparing
21:02:41 ff0839a7f7f4: Preparing
21:02:41 1a41326226dc: Waiting
21:02:41 126d08289624: Preparing
21:02:41 e6243216b73a: Waiting
21:02:41 fd812d2f0359: Preparing
21:02:41 adc9d64b09c7: Preparing
21:02:41 39a79edbb16f: Waiting
21:02:41 813c2d431507: Preparing
21:02:41 9b36ebe466d8: Waiting
21:02:41 4a18206c45bb: Preparing
21:02:41 f3462ba5b085: Waiting
21:02:41 ce33398259d3: Waiting
21:02:41 d31810a10047: Preparing
21:02:41 7b63f9d2090d: Waiting
21:02:41 1a1757b4ffe0: Preparing
21:02:41 f9c1597c6dca: Waiting
21:02:41 fd812d2f0359: Waiting
21:02:41 5624d338c0cb: Preparing
21:02:41 ff0839a7f7f4: Waiting
21:02:41 e201260e6184: Preparing
21:02:41 adc9d64b09c7: Waiting
21:02:41 1133070ab3c8: Preparing
21:02:41 126d08289624: Waiting
21:02:41 4a18206c45bb: Waiting
21:02:41 e8f010966380: Preparing
21:02:41 81c1167eafa0: Preparing
21:02:41 7fffe692204a: Preparing
21:02:41 04fff313a76c: Preparing
21:02:41 04a55213b645: Preparing
21:02:41 5624d338c0cb: Waiting
21:02:41 e201260e6184: Waiting
21:02:41 1133070ab3c8: Waiting
21:02:41 16542a8fc3be: Preparing
21:02:41 6597da2e2e52: Preparing
21:02:41 1a1757b4ffe0: Waiting
21:02:41 e8f010966380: Waiting
21:02:41 977183d4e999: Preparing
21:02:41 81c1167eafa0: Waiting
21:02:41 c8be1b8f4d60: Preparing
21:02:41 04fff313a76c: Waiting
21:02:41 16542a8fc3be: Waiting
21:02:41 977183d4e999: Waiting
21:02:41 6597da2e2e52: Waiting
21:02:41 c8be1b8f4d60: Waiting
21:02:41 06e44bb98520: Layer already exists
21:02:41 0ada10fce196: Layer already exists
21:02:41 26be1857f4d7: Layer already exists
21:02:41 e024c2664c6b: Layer already exists
21:02:41 1be4fdd704b0: Layer already exists
21:02:41 f3462ba5b085: Layer already exists
21:02:41 1a41326226dc: Layer already exists
21:02:41 e6243216b73a: Layer already exists
21:02:41 39a79edbb16f: Layer already exists
21:02:41 f9c1597c6dca: Layer already exists
21:02:41 9b36ebe466d8: Layer already exists
21:02:41 7b63f9d2090d: Layer already exists
21:02:41 ff0839a7f7f4: Layer already exists
21:02:41 126d08289624: Layer already exists
21:02:41 fd812d2f0359: Layer already exists
21:02:41 adc9d64b09c7: Layer already exists
21:02:41 ce33398259d3: Layer already exists
21:02:41 813c2d431507: Layer already exists
21:02:41 1a1757b4ffe0: Layer already exists
21:02:41 d31810a10047: Layer already exists
21:02:41 4a18206c45bb: Layer already exists
21:02:41 5624d338c0cb: Layer already exists
21:02:41 e201260e6184: Layer already exists
21:02:41 1133070ab3c8: Layer already exists
21:02:41 e8f010966380: Layer already exists
21:02:41 04fff313a76c: Layer already exists
21:02:41 7fffe692204a: Layer already exists
21:02:41 81c1167eafa0: Layer already exists
21:02:41 16542a8fc3be: Layer already exists
21:02:41 04a55213b645: Layer already exists
21:02:41 977183d4e999: Layer already exists
21:02:41 c8be1b8f4d60: Layer already exists
21:02:41 6597da2e2e52: Layer already exists
21:02:41 latest: digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f size: 7237
And a docker images
command shows latest is still the previous image (built/tagged 38 minutes ago):
21:02:41 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
21:02:41 REPOSITORY TAG IMAGE ID CREATED SIZE
21:02:41 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:337502c14082ce20c670c8a8cd53c4a505a7ee5a1a4c79d6784b35a1923700ee 38 minutes ago 6.75GB
Were there actual changes between the previous (38 minutes ago) and latest build?
Yes, the build process is triggered only on changes in the git repository.
Previous run output (albeit it had to do a docker pull to see the new version locally, check the two docker images
command output):
20:22:26 > git checkout -f ec6caf01c143c61e559623ac46ba49cd6ab518e5 # timeout=10
[...]
20:23:56 #9 writing image sha256:337502c14082ce20c670c8a8cd53c4a505a7ee5a1a4c79d6784b35a1923700ee done
20:23:56 #9 naming to AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest done
20:23:56 #9 DONE 3.8s
[...]
20:24:15 latest: digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f size: 7237
20:24:15 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins1657966881012180645.sh
20:24:15 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:15 REPOSITORY TAG IMAGE ID CREATED SIZE
20:24:15 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:236ec0b6e4ae6047dc79f19467609c4bbf160f51f0d151cf7599739eb3554078 13 minutes ago 6.75GB
20:24:15 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7728035358095848578.sh
20:24:15 + docker pull AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 latest: Pulling from om-datasci
20:24:16 Digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f
20:24:16 Status: Downloaded newer image for AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 [0 Build the om-datasci Docker image] $ /bin/sh -xe /tmp/jenkins7542804334142638886.sh
20:24:16 + docker images --no-trunc AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest
20:24:16 REPOSITORY TAG IMAGE ID CREATED SIZE
20:24:16 AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci latest sha256:337502c14082ce20c670c8a8cd53c4a505a7ee5a1a4c79d6784b35a1923700ee 24 seconds ago 6.75GB
and the new (failing) one:
21:02:40 #9 writing image sha256:fce7a99dd919ea9be1fae77dad9d534eb2bc2e2be4ec6dd1257b83d98aed9ff6 done
21:02:40 #9 naming to AWS_ID.dkr.ecr.us-west-2.amazonaws.com/om-datasci:latest done
21:02:40 #9 DONE 4.1s
[...]
21:02:41 latest: digest: sha256:ba7b2a6e1fdbebc0c8652d31c0fc76f7492c235338739c21e6baca1877f8123f size: 7237
which has a different hash, but after the push, it writes the old one.
We build docker images on an AWS EC2 instance, pushing them to AWS ECR. The pipeline is controlled by Jenkins. Amongst a lot of other things, the following happens (in this order):
docker images
command, which -in the failure situation- shows the previous image's hash.I can't see any errors.
Expected behavior
I would expect that after a successful docker image tagging, all further operations should see the new image as
latest
.Actual behavior
Copying the relevant output from the above process: 1.
2.
So according to the logs, the image was successfully tagged and uploaded to ECR (verified, however it doesn't really matters here).
The other job reports that it sees the previous image as
latest
: 4.Steps to reproduce the behavior
Basically what happens here is:
I couldn't yet find a scenario in which I could reproduce this manually.
Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.)