search

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
7 stars 2 forks source link

Failure during deploy of images caused corrupted result #5746

Closed dsotirho-ucsc closed 10 months ago

dsotirho-ucsc commented 12 months ago

A no space left on device error experienced during the deploy of the shared component did not immediately halt the deploy, and caused a corrupted result in ECR. Specifically the pycharm 2023.2.5-6-linux-arm64 image was uploaded to ECR and tagged correctly, however the 2023.2.5-6-linux-amd64 image failed to upload, and the tagging operation then tagged the existing 2023.2.5-6-linux-arm64 with an additional and incorrect 2023.2.5-6-linux-amd64 tag.

Terminal log (trimmed) ``` null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:39,007 INFO MainThread azul.docker: Distribution for image ImageRef(registry='docker.io', username='ucscgi', repository=['azul-pycharm'], tag='2023.2.5-6') … null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:42,382 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pulling from ucscgi/azul-pycharm","id":"2023.2.5-6"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:42,841 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Already exists","progressDetail":{},"id":"bbf382c14c7b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:42,848 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pulling fs layer","progressDetail":{},"id":"fe0171f85f97"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:44,391 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":178,"total":178},"progress":"[==================================================\u003e] 178B/178B","id":"93de9491f560"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Verifying Checksum","progressDetail":{},"id":"93de9491f560"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"93de9491f560"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:45,288 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":173,"total":173},"progress":"[==================================================\u003e] 173B/173B","id":"e120be579cfa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Verifying Checksum","progressDetail":{},"id":"e120be579cfa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"e120be579cfa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:46,292 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":732,"total":4099},"progress":"[========\u003e ] 732B/4.099kB","id":"177c9617d5aa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Verifying Checksum","progressDetail":{},"id":"177c9617d5aa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:46,292 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Download complete","progressDetail":{},"id":"177c9617d5aa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:47,229 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Download complete","progressDetail":{},"id":"b2452f22f136"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:51,276 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":176112011,"total":176936664},"progress":"[=================================================\u003e ] 176.1MB/176.9MB","id":"fe0171f85f97"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:40:51,310 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Download complete","progressDetail":{},"id":"fe0171f85f97"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:12,160 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":597832079,"total":598158548},"progress":"[=================================================\u003e ] 597.8MB/598.2MB","id":"c9f8c89e015d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:12,174 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Verifying Checksum","progressDetail":{},"id":"c9f8c89e015d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"c9f8c89e015d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:12,665 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":176936664,"total":176936664},"progress":"[==================================================\u003e] 176.9MB/176.9MB","id":"fe0171f85f97"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:13,528 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"fe0171f85f97"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:13,532 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":114,"total":114},"progress":"[==================================================\u003e] 114B/114B","id":"630c2afa63ac"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:13,570 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"630c2afa63ac"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,708 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":598158548,"total":598158548},"progress":"[==================================================\u003e] 598.2MB/598.2MB","id":"c9f8c89e015d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,765 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"c9f8c89e015d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,769 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":178,"total":178},"progress":"[==================================================\u003e] 178B/178B","id":"93de9491f560"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,905 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"93de9491f560"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,926 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":173,"total":173},"progress":"[==================================================\u003e] 173B/173B","id":"e120be579cfa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,936 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"e120be579cfa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,940 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":4099,"total":4099},"progress":"[==================================================\u003e] 4.099kB/4.099kB","id":"177c9617d5aa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,967 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"177c9617d5aa"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,970 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":184,"total":184},"progress":"[==================================================\u003e] 184B/184B","id":"b2452f22f136"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,982 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"b2452f22f136"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,994 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Digest: sha256:f10846f2eb2c8f21b5ce5c40fb75ca416756c0e9dc3226da4210a460e3ccbe22"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:44,996 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Status: Downloaded newer image for ucscgi/azul-pycharm:2023.2.5-6"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:45,003 INFO MainThread __main__: Pulled image ImageRef(registry='docker.io', username='ucscgi', repository=['azul-pycharm'], tag='2023.2.5-6') for platform Platform(os='linux', arch='arm64', variant=None) null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:45,059 INFO MainThread __main__: Pushing image ImageRef(registry='122796619775.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=['ucscgi', 'azul-pycharm'], tag='2023.2.5-6-linux-arm64') null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:45,329 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"The push refers to repository [122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm]"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:45,709 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Preparing","progressDetail":{},"id":"48ae5e2c5d9b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:46,653 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":3072},"progress":" 3.072kB","id":"5721b71e03fc"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:46,673 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":3072},"progress":" 3.072kB","id":"f943d79ebcd9"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:46,707 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":512,"total":24},"progress":"[==================================================\u003e] 512B","id":"48ae5e2c5d9b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:46,852 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":3468,"total":336140},"progress":"[\u003e ] 3.468kB/336.1kB","id":"a89a30f42280"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:46,952 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":16838,"total":336140},"progress":"[==\u003e ] 16.84kB/336.1kB","id":"a89a30f42280"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:46,990 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":350208,"total":336140},"progress":"[==================================================\u003e] 350.2kB","id":"a89a30f42280"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:47,681 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":3072,"total":24},"progress":"[==================================================\u003e] 3.072kB","id":"48ae5e2c5d9b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:48,318 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"5721b71e03fc"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:48,322 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"f943d79ebcd9"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:49,049 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"48ae5e2c5d9b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:49,274 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":2048},"progress":" 2.048kB","id":"ec2f3cb6748a"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:49,411 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"a89a30f42280"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:41:51,647 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"ec2f3cb6748a"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:46:36,393 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":123219968,"total":117984160},"progress":"[==================================================\u003e] 123.2MB","id":"581bf1c142e3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:46:50,207 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"581bf1c142e3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 14:54:57,193 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":458094592,"total":444650362},"progress":"[==================================================\u003e] 458.1MB","id":"d23e349ef468"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:36,433 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushing","progressDetail":{"current":1631498752,"total":1626566193},"progress":"[==================================================\u003e] 1.631GB","id":"24097762f37f"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:41,393 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Pushed","progressDetail":{},"id":"24097762f37f"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:44,605 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"2023.2.5-6-linux-arm64: digest: sha256:fdca011ec09c7b5921e670fdcb49a0ebcec0efcef7a908630263809803bcfe8d size: 1991"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:44,613 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"progressDetail":{},"aux":{"Tag":"2023.2.5-6-linux-arm64","Digest":"sha256:fdca011ec09c7b5921e670fdcb49a0ebcec0efcef7a908630263809803bcfe8d","Size":1991}} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:44,627 INFO MainThread __main__: Pushed image ImageRef(registry='122796619775.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=['ucscgi', 'azul-pycharm'], tag='2023.2.5-6-linux-arm64') null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:44,632 INFO MainThread __main__: Pulling image ImageRef(registry='docker.io', username='ucscgi', repository=['azul-pycharm'], tag='2023.2.5-6') for platform Platform(os='linux', arch='amd64', variant=None) … null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:45,897 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pulling from ucscgi/azul-pycharm","id":"2023.2.5-6"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:46,281 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Already exists","progressDetail":{},"id":"d1da99c2f148"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:46,288 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pulling fs layer","progressDetail":{},"id":"ffc6e82553a3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:46,289 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pulling fs layer","progressDetail":{},"id":"1adc0e7ce5df"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:46,984 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":114,"total":114},"progress":"[==================================================\u003e] 114B/114B","id":"87093952c797"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Verifying Checksum","progressDetail":{},"id":"87093952c797"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"87093952c797"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:47,873 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":177,"total":177},"progress":"[==================================================\u003e] 177B/177B","id":"0c35da56dbdf"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"0c35da56dbdf"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:48,738 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Download complete","progressDetail":{},"id":"7fd66165be54"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:49,580 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":733,"total":4090},"progress":"[========\u003e ] 733B/4.09kB","id":"37fa53bc699d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:49,581 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":4090,"total":4090},"progress":"[==================================================\u003e] 4.09kB/4.09kB","id":"37fa53bc699d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Verifying Checksum","progressDetail":{},"id":"37fa53bc699d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"37fa53bc699d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:05:50,481 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":184,"total":184},"progress":"[==================================================\u003e] 184B/184B","id":"1adc0e7ce5df"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Verifying Checksum","progressDetail":{},"id":"1adc0e7ce5df"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"1adc0e7ce5df"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:00,285 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":178491956,"total":181586222},"progress":"[=================================================\u003e ] 178.5MB/181.6MB","id":"ffc6e82553a3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:00,385 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Verifying Checksum","progressDetail":{},"id":"ffc6e82553a3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"status":"Download complete","progressDetail":{},"id":"ffc6e82553a3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:13,567 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Downloading","progressDetail":{"current":591848323,"total":594343085},"progress":"[=================================================\u003e ] 591.8MB/594.3MB","id":"8ba46c160c3d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:13,617 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Download complete","progressDetail":{},"id":"8ba46c160c3d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:20,133 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":181586222,"total":181586222},"progress":"[==================================================\u003e] 181.6MB/181.6MB","id":"ffc6e82553a3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:20,283 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"ffc6e82553a3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:20,287 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":114,"total":114},"progress":"[==================================================\u003e] 114B/114B","id":"87093952c797"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:20,298 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Pull complete","progressDetail":{},"id":"87093952c797"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:37,422 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":497451008,"total":594343085},"progress":"[=========================================\u003e ] 497.5MB/594.3MB","id":"8ba46c160c3d"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:37,542 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"status":"Extracting","progressDetail":{"current":594343085,"total":594343085},"progress":"[==================================================\u003e] 594.3MB/594.3MB","id":"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:37,549 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker pull {"errorDetail":{"message":"failed to register layer: write /opt/pycharm/lib/grpc.jar: no space left on device"},"error":"failed to register layer: write /opt/pycharm/lib/grpc.jar: no space left on device"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:37,551 INFO MainThread __main__: Pulled image ImageRef(registry='docker.io', username='ucscgi', repository=['azul-pycharm'], tag='2023.2.5-6') for platform Platform(os='linux', arch='amd64', variant=None) null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:37,914 INFO MainThread __main__: Pushing image ImageRef(registry='122796619775.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=['ucscgi', 'azul-pycharm'], tag='2023.2.5-6-linux-amd64') null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,110 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"The push refers to repository [122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm]"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,480 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Preparing","progressDetail":{},"id":"48ae5e2c5d9b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,482 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Preparing","progressDetail":{},"id":"a89a30f42280"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,881 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"5721b71e03fc"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,895 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"f943d79ebcd9"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,905 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"48ae5e2c5d9b"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,908 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"24097762f37f"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:38,920 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"a89a30f42280"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:39,287 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"d23e349ef468"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:39,289 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"ec2f3cb6748a"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:39,304 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"Layer already exists","progressDetail":{},"id":"581bf1c142e3"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:40,171 INFO MainThread __main__: docker.io/ucscgi/azul-pycharm:2023.2.5-6: docker push {"status":"2023.2.5-6-linux-amd64: digest: sha256:fdca011ec09c7b5921e670fdcb49a0ebcec0efcef7a908630263809803bcfe8d size: 1991"} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): {"progressDetail":{},"aux":{"Tag":"2023.2.5-6-linux-amd64","Digest":"sha256:fdca011ec09c7b5921e670fdcb49a0ebcec0efcef7a908630263809803bcfe8d","Size":1991}} null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:40,173 INFO MainThread __main__: Pushed image ImageRef(registry='122796619775.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=['ucscgi', 'azul-pycharm'], tag='2023.2.5-6-linux-amd64') null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:40,174 INFO MainThread __main__: Creating manifest image ImageRef(registry='122796619775.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=['ucscgi', 'azul-pycharm'], tag='2023.2.5-6') null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): No such manifest: 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm:2023.2.5-6 null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): Created manifest list 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm:2023.2.5-6 null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): sha256:32281f14b22fdde8688ffa045108889c62f8baf43b3567728f243a42988114d6 null_resource.image_fcz_J8X7-rWV8eJhj-_w2lYBnNg (local-exec): 2023-11-29 15:06:44,740 INFO MainThread __main__: Created manifest image ImageRef(registry='122796619775.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=['ucscgi', 'azul-pycharm'], tag='2023.2.5-6') null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): 2023-11-29 15:13:49,366 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull {"status":"Downloading","progressDetail":{"current":1086830909,"total":1314564115},"progress":"[=========================================\u003e ] 1.087GB/1.315GB","id":"a12840f5230e"} null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): 2023-11-29 15:13:49,484 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull {"status":"Downloading","progressDetail":{"current":1090050365,"total":1314564115},"progress":"[=========================================\u003e ] 1.09GB/1.315GB","id":"a12840f5230e"} null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): 2023-11-29 15:13:49,510 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull {"status":"Downloading","progressDetail":{"current":1314564115,"total":1314564115},"progress":"[==================================================\u003e] 1.315GB/1.315GB","id":"a12840f5230e"} null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): 2023-11-29 15:13:49,757 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull {"errorDetail":{"message":"write /var/lib/docker/tmp/GetImageBlob2790274496: no space left on device"},"error":"write /var/lib/docker/tmp/GetImageBlob2790274496: no space left on device"} null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): 2023-11-29 15:13:49,759 INFO MainThread __main__: Pulled image ImageRef(registry='docker.io', username='gitlab', repository=['gitlab-ce'], tag='16.6.0-ce.0') for platform Platform(os='linux', arch='amd64', variant=None) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): 2023-11-29 15:13:49,775 INFO MainThread __main__: Released semaphore '/copy_images_to_ecr.dev' null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): Traceback (most recent call last): null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): response.raise_for_status() null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): raise HTTPError(http_error_msg, response=self) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-ce:16.6.0-ce.0/json null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): The above exception was the direct cause of the following exception: null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): Traceback (most recent call last): null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 234, in null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): main() null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 220, in main null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): copy_image(image) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 59, in copy_image null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): image = client.images.get(str(src)) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/models/images.py", line 335, in get null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): return self.prepare_model(self.client.api.inspect_image(name)) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): return f(self, resource_id, *args, **kwargs) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): return self._result( null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): ^^^^^^^^^^^^^ null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): self._raise_for_status(response) null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): raise create_api_error_from_http_exception(e) from e null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): raise cls(e, response=response, explanation=explanation) from e null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw (local-exec): docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-ce:16.6.0-ce.0/json: Not Found ("No such image: gitlab/gitlab-ce:16.6.0-ce.0") null_resource.image_kpwcsza8I6qXXqcevs5CCohOTaE (local-exec): 2023-11-29 15:13:49,803 INFO MainThread __main__: Acquired semaphore '/copy_images_to_ecr.dev' null_resource.image_kpwcsza8I6qXXqcevs5CCohOTaE (local-exec): 2023-11-29 15:13:49,933 INFO MainThread azul.docker: Distribution for image ImageRef(registry='docker.io', username='clamav', repository=['clamav'], tag='1.2.1-17') … null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI: Still creating... [45m21s elapsed] null_resource.image_kpwcsza8I6qXXqcevs5CCohOTaE: Still creating... [45m21s elapsed] \ null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:46,786 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull {"status":"Extracting","progressDetail":{"current":506920960,"total":537741875},"progress":"[===============================================\u003e ] 506.9MB/537.7MB","id":"92a02b79ebd0"} null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:46,889 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull {"status":"Extracting","progressDetail":{"current":517505024,"total":537741875},"progress":"[================================================\u003e ] 517.5MB/537.7MB","id":"92a02b79ebd0"} null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:46,990 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull {"status":"Extracting","progressDetail":{"current":528646144,"total":537741875},"progress":"[=================================================\u003e ] 528.6MB/537.7MB","id":"92a02b79ebd0"} null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:47,159 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull {"status":"Extracting","progressDetail":{"current":537741875,"total":537741875},"progress":"[==================================================\u003e] 537.7MB/537.7MB","id":"92a02b79ebd0"} null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:47,162 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull {"errorDetail":{"message":"failed to register layer: write /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz: no space left on device"},"error":"failed to register layer: write /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz: no space left on device"} null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:47,164 INFO MainThread __main__: Pulled image ImageRef(registry='docker.io', username='gitlab', repository=['gitlab-runner'], tag='ubuntu-v16.6.1') for platform Platform(os='linux', arch='arm64', variant=None) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): 2023-11-29 15:14:47,186 INFO MainThread __main__: Released semaphore '/copy_images_to_ecr.dev' null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): Traceback (most recent call last): null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): response.raise_for_status() null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): raise HTTPError(http_error_msg, response=self) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1/json null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): The above exception was the direct cause of the following exception: null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): Traceback (most recent call last): null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 234, in null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): main() null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 220, in main null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): copy_image(image) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 59, in copy_image null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): image = client.images.get(str(src)) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/models/images.py", line 335, in get null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): return self.prepare_model(self.client.api.inspect_image(name)) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): return f(self, resource_id, *args, **kwargs) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): return self._result( null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): ^^^^^^^^^^^^^ null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): self._raise_for_status(response) null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): raise create_api_error_from_http_exception(e) from e null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): raise cls(e, response=response, explanation=explanation) from e null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI (local-exec): docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1/json: Not Found ("No such image: gitlab/gitlab-runner:ubuntu-v16.6.1") ╷ │ Error: local-exec provisioner error │ │ with null_resource.image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw, │ on shared.tf.json line 1499, in resource[36].null_resource[5].image_ATHoVdQFbvLsbBX6RzuQ9K8ouNw.provisioner.local-exec: │ 1499: } │ │ Error running command 'python /Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py --copy docker.io/gitlab/gitlab-ce:16.6.0-ce.0': exit status 1. Output: │ otal":1314564115},"progress":"[=======================================\u003e ] 1.038GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:47,835 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1041307965,"total":1314564115},"progress":"[=======================================\u003e ] │ 1.041GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:47,943 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1044523325,"total":1314564115},"progress":"[=======================================\u003e ] │ 1.045GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,051 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1047746877,"total":1314564115},"progress":"[=======================================\u003e ] │ 1.048GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,159 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1050958141,"total":1314564115},"progress":"[=======================================\u003e ] │ 1.051GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,266 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1054169405,"total":1314564115},"progress":"[========================================\u003e ] │ 1.054GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,378 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1057384765,"total":1314564115},"progress":"[========================================\u003e ] │ 1.057GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,488 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1060059453,"total":1314564115},"progress":"[========================================\u003e ] │ 1.06GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,596 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1063799101,"total":1314564115},"progress":"[========================================\u003e ] │ 1.064GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,698 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1067002173,"total":1314564115},"progress":"[========================================\u003e ] │ 1.067GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,806 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1070209341,"total":1314564115},"progress":"[========================================\u003e ] │ 1.07GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:48,925 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1073424701,"total":1314564115},"progress":"[========================================\u003e ] │ 1.073GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,043 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1077205309,"total":1314564115},"progress":"[========================================\u003e ] │ 1.077GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,152 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1080412477,"total":1314564115},"progress":"[=========================================\u003e ] │ 1.08GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,257 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1083619645,"total":1314564115},"progress":"[=========================================\u003e ] │ 1.084GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,366 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1086830909,"total":1314564115},"progress":"[=========================================\u003e ] │ 1.087GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,484 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1090050365,"total":1314564115},"progress":"[=========================================\u003e ] │ 1.09GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,510 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull │ {"status":"Downloading","progressDetail":{"current":1314564115,"total":1314564115},"progress":"[==================================================\u003e] │ 1.315GB/1.315GB","id":"a12840f5230e"} │ 2023-11-29 15:13:49,757 INFO MainThread __main__: docker.io/gitlab/gitlab-ce:16.6.0-ce.0: docker pull {"errorDetail":{"message":"write │ /var/lib/docker/tmp/GetImageBlob2790274496: no space left on device"},"error":"write /var/lib/docker/tmp/GetImageBlob2790274496: no space left on device"} │ 2023-11-29 15:13:49,759 INFO MainThread __main__: Pulled image ImageRef(registry='docker.io', username='gitlab', repository=['gitlab-ce'], tag='16.6.0-ce.0') for │ platform Platform(os='linux', arch='amd64', variant=None) │ 2023-11-29 15:13:49,775 INFO MainThread __main__: Released semaphore '/copy_images_to_ecr.dev' │ Traceback (most recent call last): │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status │ response.raise_for_status() │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status │ raise HTTPError(http_error_msg, response=self) │ requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-ce:16.6.0-ce.0/json │ │ The above exception was the direct cause of the following exception: │ │ Traceback (most recent call last): │ File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 234, in │ main() │ File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 220, in main │ copy_image(image) │ File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 59, in copy_image │ image = client.images.get(str(src)) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/models/images.py", line 335, in get │ return self.prepare_model(self.client.api.inspect_image(name)) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped │ return f(self, resource_id, *args, **kwargs) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image │ return self._result( │ ^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result │ self._raise_for_status(response) │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status │ raise create_api_error_from_http_exception(e) from e │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception │ raise cls(e, response=response, explanation=explanation) from e │ docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-ce:16.6.0-ce.0/json: Not Found ("No such image: │ gitlab/gitlab-ce:16.6.0-ce.0") │ ╵ ╷ │ Error: local-exec provisioner error │ │ with null_resource.image_FkWnQqCMAbHCnQuSp0oGylCrTJI, │ on shared.tf.json line 1519, in resource[36].null_resource[6].image_FkWnQqCMAbHCnQuSp0oGylCrTJI.provisioner.local-exec: │ 1519: } │ │ Error running command 'python /Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py --copy docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1': exit status 1. Output: │ b-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":365428736,"total":537741875},"progress":"[=================================\u003e ] │ 365.4MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:45,559 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":376569856,"total":537741875},"progress":"[===================================\u003e ] │ 376.6MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:45,662 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":387710976,"total":537741875},"progress":"[====================================\u003e ] │ 387.7MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:45,765 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":398852096,"total":537741875},"progress":"[=====================================\u003e ] │ 398.9MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:45,869 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":409993216,"total":537741875},"progress":"[======================================\u003e ] │ 410MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:45,972 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":420577280,"total":537741875},"progress":"[=======================================\u003e ] │ 420.6MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,073 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":431161344,"total":537741875},"progress":"[========================================\u003e ] │ 431.2MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,176 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":442302464,"total":537741875},"progress":"[=========================================\u003e ] │ 442.3MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,277 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":453443584,"total":537741875},"progress":"[==========================================\u003e ] │ 453.4MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,380 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":464584704,"total":537741875},"progress":"[===========================================\u003e ] │ 464.6MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,483 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":475725824,"total":537741875},"progress":"[============================================\u003e ] │ 475.7MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,584 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":485752832,"total":537741875},"progress":"[=============================================\u003e ] │ 485.8MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,686 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":496336896,"total":537741875},"progress":"[==============================================\u003e ] │ 496.3MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,786 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":506920960,"total":537741875},"progress":"[===============================================\u003e ] │ 506.9MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,889 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":517505024,"total":537741875},"progress":"[================================================\u003e ] │ 517.5MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:46,990 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":528646144,"total":537741875},"progress":"[=================================================\u003e ] │ 528.6MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:47,159 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull │ {"status":"Extracting","progressDetail":{"current":537741875,"total":537741875},"progress":"[==================================================\u003e] │ 537.7MB/537.7MB","id":"92a02b79ebd0"} │ 2023-11-29 15:14:47,162 INFO MainThread __main__: docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1: docker pull {"errorDetail":{"message":"failed to register layer: │ write /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz: no space left on device"},"error":"failed to register layer: write │ /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz: no space left on device"} │ 2023-11-29 15:14:47,164 INFO MainThread __main__: Pulled image ImageRef(registry='docker.io', username='gitlab', repository=['gitlab-runner'], │ tag='ubuntu-v16.6.1') for platform Platform(os='linux', arch='arm64', variant=None) │ 2023-11-29 15:14:47,186 INFO MainThread __main__: Released semaphore '/copy_images_to_ecr.dev' │ Traceback (most recent call last): │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 268, in _raise_for_status │ response.raise_for_status() │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status │ raise HTTPError(http_error_msg, response=self) │ requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1/json │ │ The above exception was the direct cause of the following exception: │ │ Traceback (most recent call last): │ File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 234, in │ main() │ File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 220, in main │ copy_image(image) │ File "/Users/daniel/repo/azul1/scripts/copy_images_to_ecr.py", line 59, in copy_image │ image = client.images.get(str(src)) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/models/images.py", line 335, in get │ return self.prepare_model(self.client.api.inspect_image(name)) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/utils/decorators.py", line 19, in wrapped │ return f(self, resource_id, *args, **kwargs) │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/image.py", line 251, in inspect_image │ return self._result( │ ^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 274, in _result │ self._raise_for_status(response) │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/api/client.py", line 270, in _raise_for_status │ raise create_api_error_from_http_exception(e) from e │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ │ File "/Users/daniel/repo/azul1/.venv/lib/python3.11/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception │ raise cls(e, response=response, explanation=explanation) from e │ docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.43/images/docker.io/gitlab/gitlab-runner:ubuntu-v16.6.1/json: Not Found ("No such image: │ gitlab/gitlab-runner:ubuntu-v16.6.1") │ ╵ make: *** [apply] Error 1 (.venv) daniel@Crispin ~/repo/azul1 $ ```

terminal-log-full.txt

$ aws ecr describe-images --repository-name docker.io/ucscgi/azul-pycharm
{
    "imageDetails": [
        {
            "registryId": "122796619775",
            "repositoryName": "docker.io/ucscgi/azul-pycharm",
            "imageDigest": "sha256:32281f14b22fdde8688ffa045108889c62f8baf43b3567728f243a42988114d6",
            "imageTags": [
                "2023.2.5-6"
            ],
            "imageSizeInBytes": 828813127,
            "imagePushedAt": 1701299205.0,
            "imageManifestMediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
            "lastRecordedPullTime": 1701315930.874
        },
        {
            "registryId": "122796619775",
            "repositoryName": "docker.io/ucscgi/azul-pycharm",
            "imageDigest": "sha256:fdca011ec09c7b5921e670fdcb49a0ebcec0efcef7a908630263809803bcfe8d",
            "imageTags": [
                "2023.2.5-6-linux-arm64",
                "2023.2.5-6-linux-amd64"
            ],
            "imageSizeInBytes": 828813127,
            "imagePushedAt": 1701299145.0,
            "imageManifestMediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "artifactMediaType": "application/vnd.docker.container.image.v1+json",
            "lastRecordedPullTime": 1701299201.345
        }
    ]
}
dsotirho-ucsc commented 12 months ago

Assignee to expand on description.

dsotirho-ucsc commented 11 months ago

Spike to diagnose.

hannes-ucsc commented 11 months ago

The Docker SDK and underlying engine API are basically crap when it comes to pulling platform-specific images and detecting failure. The comments in the patch below elaborate on this some more. The patch should ensure that we reliably detect failed pulls.

Index: scripts/copy_images_to_ecr.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/scripts/copy_images_to_ecr.py b/scripts/copy_images_to_ecr.py
--- a/scripts/copy_images_to_ecr.py (revision 0bd9865570a1d2f39cdc5d363ffb82ac1445d799)
+++ b/scripts/copy_images_to_ecr.py (date 1701826403204)
@@ -22,6 +22,7 @@
 from azul import (
     config,
     reject,
+    require,
 )
 from azul.args import (
     AzulArgumentHelpFormatter,
@@ -50,13 +51,33 @@
     client = docker.client.from_env()
     for platform in src.platforms:
         log.info('Pulling image %r for platform %r …', src, platform)
+        # We're at liberty to pick the destination tag, and can therefore make
+        # it platform-specific, the source tag is often ambiguous in that it may
+        # match multiple images for different platforms. The `platform` argument
+        # to `pull` disambiguates that, and ensures that the right image is
+        # pulled, but the pull also tags the resulting image locally using the
+        # given ambiguous tag. This means that a pull for a given tag and
+        # platform may clobber the tag from an earlier pull of the same tag but
+        # another platform. Furthermore, it is difficult to detect a failed pull
+        # because one would have to look for the absence of the output line
+        # indicating success. The format of that line is not documented.
         output = client.api.pull(src.name,
                                  platform=str(platform),
                                  tag=src.tag,
                                  stream=True)
         log_lines(src, 'pull', output)
         log.info('Pulled image %r for platform %r', src, platform)
+        # For ambiguous tags the following line will get the image for the most
+        # recent *successful* pull. If the most recent pull failed, this will
+        # get the image from the successful pull before that. If no pulls were
+        # successful for this tag, the line will fail.
         image = client.images.get(str(src))
+        # If there was a previous successful pull for the tag, the assertions
+        # below will fail.
+        require(image.attrs['Architecture'] == platform.arch,
+                'Pull failed, local image has wrong architecture)', image.attrs)
+        require(image.attrs['Os'] == platform.os,
+                'Pull failed, local image has wrong OS)', image.attrs)
         dst_part = ImageRef.create(name=config.docker_registry + src.name,
                                    tag=make_platform_tag(src.tag, platform))
         image.tag(dst_part.name, tag=dst_part.tag)
hannes-ucsc commented 11 months ago

Hard to reproduce. I'm happy to accept this as a tentative fix and reiterate if issue persists.