search

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
6 stars 2 forks source link

Ambiguous errors attempting to deploy shared component with viewer role #6231

Open dsotirho-ucsc opened 4 months ago

dsotirho-ucsc commented 4 months ago 
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): Executing: ["/bin/sh" "-c" "python /Users/daniel/repo/azul3/scripts/manage_images.py --copy docker.io/gitlab/gitlab-ce:16.11.1-ce.0"]
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:17,316   DEBUG MainThread __main__: Creating or opening semaphore '/manage_images.prod'
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:17,316    INFO MainThread __main__: Acquiring semaphore '/manage_images.prod'
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,259    INFO MainThread __main__: Acquired semaphore '/manage_images.prod'
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,259    INFO MainThread __main__: Copying image TagImageRef(registry='docker.io', username='gitlab', repository=('gitlab-ce',), tag='16.11.1-ce.0') for platform Platform(os='linux', arch='amd64', variant=None)
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,259    INFO MainThread azul.docker: Pulling image DigestImageRef(registry='docker.io', username='gitlab', repository=('gitlab-ce',), digest='sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb') …
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,650   DEBUG MainThread azul.docker: docker.io/gitlab/gitlab-ce@sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb: docker pull {"status":"Pulling from gitlab/gitlab-ce","id":"docker.io/gitlab/gitlab-ce@sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,652   DEBUG MainThread azul.docker: docker.io/gitlab/gitlab-ce@sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb: docker pull {"status":"Digest: sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Status: Image is up to date for gitlab/gitlab-ce@sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,659    INFO MainThread azul.docker: Pulled image DigestImageRef(registry='docker.io', username='gitlab', repository=('gitlab-ce',), digest='sha256:9f80ba264bd3dce7b73fa6d70f41552669405f6ed129518c64d6d373333d77cb')
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:20,695    INFO MainThread azul.docker: Pushing image TagImageRef(registry='542754589326.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=('gitlab', 'gitlab-ce'), tag='16.11.1-ce.0') …
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:21,864   DEBUG MainThread azul.docker: 542754589326.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-ce:16.11.1-ce.0: docker push {"status":"The push refers to repository [542754589326.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-ce]"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:22,243   DEBUG MainThread azul.docker: 542754589326.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-ce:16.11.1-ce.0: docker push {"status":"Preparing","progressDetail":{},"id":"843453802e9c"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"37070dc9850e"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"03604535d761"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"08bb549d56b4"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"887e838b8879"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"b292e22baba0"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"178b6a69880c"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"f5716f9644bb"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Preparing","progressDetail":{},"id":"e0a9f5911802"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Waiting","progressDetail":{},"id":"08bb549d56b4"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Waiting","progressDetail":{},"id":"887e838b8879"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Waiting","progressDetail":{},"id":"b292e22baba0"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Waiting","progressDetail":{},"id":"178b6a69880c"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Waiting","progressDetail":{},"id":"f5716f9644bb"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): {"status":"Waiting","progressDetail":{},"id":"e0a9f5911802"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:23,031   DEBUG MainThread azul.docker: 542754589326.dkr.ecr.us-east-1.amazonaws.com/docker.io/gitlab/gitlab-ce:16.11.1-ce.0: docker push {"errorDetail":{"message":"denied: User: arn:aws:sts::542754589326:assumed-role/viewer/dsotirho@ucsc.edu is not authorized to perform: ecr:InitiateLayerUpload on resource: arn:aws:ecr:us-east-1:542754589326:repository/docker.io/gitlab/gitlab-ce because no identity-based policy allows the ecr:InitiateLayerUpload action"},"error":"denied: User: arn:aws:sts::542754589326:assumed-role/viewer/dsotirho@ucsc.edu is not authorized to perform: ecr:InitiateLayerUpload on resource: arn:aws:ecr:us-east-1:542754589326:repository/docker.io/gitlab/gitlab-ce because no identity-based policy allows the ecr:InitiateLayerUpload action"}
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:23,032    INFO MainThread azul.docker: Pushed image TagImageRef(registry='542754589326.dkr.ecr.us-east-1.amazonaws.com', username='docker.io', repository=('gitlab', 'gitlab-ce'), tag='16.11.1-ce.0')
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): 2024-05-02 18:11:23,041    INFO MainThread __main__: Released semaphore '/manage_images.prod'
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): Traceback (most recent call last):
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/scripts/manage_images.py", line 64, in copy_image
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     manifests = cast(ManifestList, manifest_or_list)['manifests']
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): KeyError: 'manifests'

null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): During handling of the above exception, another exception occurred:

null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): Traceback (most recent call last):
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/.venv/lib/python3.11/site-packages/more_itertools/more.py", line 539, in one
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     first_value = next(it)
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):                   ^^^^^^^^
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): StopIteration

null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): The above exception was the direct cause of the following exception:

null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): Traceback (most recent call last):
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/scripts/manage_images.py", line 330, in <module>
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     main()
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/scripts/manage_images.py", line 314, in main
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     copy_image(image)
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/scripts/manage_images.py", line 66, in copy_image
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     copy_single_platform_image(src, cast(Manifest, manifest_or_list), tag=src.tag)
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/scripts/manage_images.py", line 123, in copy_single_platform_image
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     return one(refs)
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):            ^^^^^^^^^
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):   File "/Users/daniel/repo/azul3/.venv/lib/python3.11/site-packages/more_itertools/more.py", line 541, in one
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec):     raise (
null_resource.image_XjUqB-QI7ittxKZxCMAvdh8U_GY (local-exec): ValueError: too few items in iterable (expected 1)
dsotirho-ucsc commented 4 months ago

Assignee to consider next steps.

hannes-ucsc commented 1 month ago

It appears that the call to push_docker_image with viewer permissions yields a value whose .attrs['RepoDigests'] is either empty or doesn't return any matching digests. I believe the latter is more likely. Spike to confirm by debugging such a call. Probably easiest to attach the PyCharm debugger to a Python console and make the call there.

However we have higher priorities so, we'll spike later.