sregistry pull from AWS ECR fails

[tbugfinder]

Please note this is probably related to https://github.com/sylabs/singularity/issues/2059

Version of sregistry:

sregistry 0.0.94 (pip)

Apparently it doesn't change after cloning this repository (version = "0.0.94").

Expected behavior

sregistry pull should pull images from AWS ECR successfully.

Actual behavior

sregistry pull fails with error message ERROR Unrecognized authentication challenge, exiting.. I've configured SREGISTRY_DOCKERHUB_BASE (with and without), SREGISTRY_DOCKER_PASSWORD and SREGISTRY_DOCKER_USERNAME with output of aws ecr get-login --no-include-email.

# sregistry pull docker://<accountid>.dkr.ecr.<region>.amazonaws.com/<path,image>:tag
ERROR Unrecognized authentication challenge, exiting.
ERROR Unrecognized authentication challenge, exiting.
ERROR Unrecognized authentication challenge, exiting.

Steps to reproduce behavior

• set SREGISTRY_DOCKER_PASSWORD and SREGISTRY_DOCKER_USERNAME
• run sregistry pull docker://< aws ecr >/

[vsoch]

hey @tbugfinder ! I don't know much about the aws docker endpoint, but my guess is that there is a different authentication workflow than is used for the others (if it's failing) could you give me a curl call that you are using that successfully works and I'll see what I can do for you? Thanks!

[tbugfinder]

AWS documentation is quite detailed - please see https://docs.aws.amazon.com/AmazonECR/latest/userguide/Registries.html

curl -i -H "Authorization: Basic $TOKEN" https://012345678910.dkr.ecr.us-east-1.amazonaws.com/v2/amazonlinux/tags/list

Is this sufficient or do you need additional details ?

[tbugfinder]

Does sregistry evaluate https_proxy settings ?

[vsoch]

Thanks! I'll need a curl command (from you) that you have tested and verified working. The documentation is great, but I need assurance I am testing something that is actually working, and not something that should be working given a documentation base.

[vsoch]

I don't think it does anything with respect to any environment variable HTTPS_PROXY but I believe users in the past interacting with sregistry-cli (via python) have just set this variable and requests discovers it automatically.

[tbugfinder]

I've verified the AWS documentation copy/paste => success. It's valid.

> Accept: */*
> Authorization: Basic kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkdjdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8
...
< Docker-Distribution-Api-Version: registry/2.0
Docker-Distribution-Api-Version: registry/2.0
< Content-Length: 165
Content-Length: 165
< Connection: Close
Connection: Close

[tbugfinder]

docker.io response header: Www-Authenticate: Bearer Realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/busybox:pull"

ecr.amazonaws.com responce header (response to curl request): Www-Authenticate: Basic realm="https://<accountid>.dkr.ecr.<region>.amazonaws.com/",service="ecr.amazonaws.com"

[vsoch]

Awesome, thank you! I'm working on some CI for another project atm but I'll try to take a look at this later this evening, and let you know if I have more questions. I've never used any container registries provided by AWS so I'm hoping I don't need to sign up / pay additionally to just test the http API.

[tbugfinder]

thx, I could probably create & share a temporary RO keypair. My understanding is now that each request should add the Basic authentication header utilizing the TOKEN (see Amazon docs).

[vsoch]

Could you please provide a (universal) container that any user can access? I am getting 403, and I have full permissions for my IAM user for the Container Registry.

[vsoch]

$ curl -i -H "Authorization: Basic $AWS_TOKEN" https://012345678910.dkr.ec
r.us-east-1.amazonaws.com/v2/amazonlinux/tags/list
HTTP/1.1 403 Forbidden
Content-Type: application/json; charset=utf-8
Date: Sat, 22 Sep 2018 22:37:00 GMT
Docker-Distribution-Api-Version: registry/2.0
Content-Length: 206
Connection: keep-alive

{"errors":[{"code":"DENIED","message":"User: arn:aws:iam::xxxxxxxxxxxxx:user/xxxxxxxx is not authorized to perform: ecr:ListImages on resource: arn:aws:ecr:us-east-1:012345678910:repository/amazonlinux"}]}

I'm also keeping notes (and will publish with the docs if I figure this out) for the complete setup and generation of credentials, so others can easily reproduce this workflow.

[vsoch]

Maybe it's about the policies? I added all the ones I could find for ContainerRegistry ContainerService and still denied. Let me know what you figure out!

[vsoch]

I'd like to write up instructions that ask for the fewest permissions as possible (as opposed to telling the user to create an IAM with full access to everything which is not so great, haha).

[vsoch]

hey @tbugfinder so now we have two ways to go about debugging this! I'm hoping that the aws team can respond to the issue, and (maybe even add a note their docs!) about the minimum set of permissions to not get a 403. In the meantime, if you figure this out, please share so I can reproduce, update the docs I'm working on, and then get to debugging the call in sregistry. My instinct is that it likely comes down to a silly missing / malformed header or similar, and we just need to cleanly reproduce the (working) curl call in python as a fix. Thanks!

[tbugfinder]

I'd start off AWS documentation. https://docs.aws.amazon.com/AmazonECR/latest/userguide/RepositoryPolicyExamples.html Didn't have a chance yet to do it. However my credentials successfully pull images using docker.

[tbugfinder]

Also the user guide is opensourced in case it has to be more detailed or corrected: https://github.com/awsdocs/amazon-ecr-user-guide. Focus of this issue s more about fixing sregistry pull <AWS ECR>.

[vsoch]

Yes, that's been where I've been reading, I can't find the answer. And yes, the issue is about getting sregistry pull to work. I'm the lead developer of sregistry client, and in order to fix this for you I need a simple use case - with curl. The request is agnostic to the technology backend. Starting with a simpler form is good practice. If you have a clever way of fixing an error starting at a higher level of abstraction or complexity, I'm all ears! If the documentation doesn't clearly show this (on the AWS side) this needs to be fixed.

[tbugfinder]

Have you tried attaching the minimal AWS managed policy? https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html#AmazonEC2ContainerRegistryReadOnly It already includes ecr:GetAuthorizationToken.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage"
            ],
            "Resource": "*"
        }
    ]
}

Any difference to your policy?

[vsoch]

Here is what I have - the second chunk are from the group:

but the FullAccess should cover everything, no?

[vsoch]

I added the permission (ReadOnly) to my "Attached Directly" group - same error here.

[tbugfinder]

Which command are you executing? Could you verify your env settings of AWS_SECRET_ACCESS_KEY and AWS_SECRET_ACCESS_KEY and maybe AWS_DEFAULT_REGION? Could you execute aws --debug ecr describe-repositories ?

[vsoch]

yes! Is there anything secret in that output (I don't see anything on first glance) so I can't post it here?

[vsoch]

My understanding is that I don't need those specific environment variables exported if I provided them via aws configure because then the code to generate the token takes this into account?

[tbugfinder]

Does aws --debug ecr describe-repositories return successfully?

If yes, does aws ecr get-login return successfully? It should display a docker login command which could you copy paste into shell. It also shows you the USERNAME and PASSWORD which are input values for sregistry pull.

[vsoch]

Yes it does! But I think this is just another avenue to getting the same token header, in the example I posted I am doing:

AWS_TOKEN=$(aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken')

The token is then added as a header:

curl -i -H "Authorization: Basic $AWS_TOKEN" https://012345678910.dkr.ecr.us-east-1.amazonaws.com/v2/amazonlinux/tags/list

The part that seems to be missing that prints from the docker login command) is this

-e none https://692517157806.dkr.ecr.us-east-1.amazonaws.com

so I tried adjusting my curl call slightly:

curl -i -H "Authorization: Basic $AWS_TOKEN" https://692517157806.dkr.ecr.us-east-1.amazonaws.com/v2

Since I don't have repos I am just testing the base. This returns 200

HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8
Date: Mon, 24 Sep 2018 19:53:16 GMT
Docker-Distribution-Api-Version: registry/2.0
Content-Length: 0
Connection: keep-alive

So it is about permissions. The example that aws docs have posted does not work out of the box without some additional permission - either that condition should be taken away, or the user instructed how to add it. Otherwise we get here where we follow an instruction and it doesn't work.

[vsoch]

And I forgot to say (in case it's not obvious!) I think the differences in URL come down to my repository vs. the one that the aws docs are mentioning, which I'm assuming is a shared amazon one, or similar?

[tbugfinder]

I don't get your point, that AWS docs lack information (/v2/amazonlinux/tags is about a specific image which you don't have). Are you now read to start with investigatng sregistry pull docker://ECR or do you need additional information in regards to AWS which I could maybe help with?

[vsoch]

the URL BEFORE that is for a particular repository (note the field that I pointed out). Their docs provide a specific repository, and it's not logical that it isn't one that a user (testing the docs) couldn't access.

sregistry pull needs to issue these exact commands, albeit in python. The key to solving this bug is to solve the basic http requests first. I hope I have made myself clear.

[vsoch]

sregistry pull docker:// uses the docker specification for the registry, and it was developed in this exact same way but with those docs. In a perfect world those two bases would overlap since aws is implementing "the same" protocol, but since it's broken this obviously isn't the case. From past cases I can almost assure you it has to do with additional headers / proxy / unexpected additions that the aws implementation has. And like I just said, the only way to figure this out to make an sregistry pull endpoint work is to debug that. If you have constructive ideas for how to work toward that goal, let's talk about that. Telling me that we need to work on sregistry pull is, well, not useful. I know this.

[vsoch]

To directly answer your question, I can definitely stop caring about if the aws docs work or not. What I absolutely need is just one container to test, with a request example with curl, and response with 200, that is successful. I'd like to be able to provide our (sregistry cli) users with documentation for setting up their credentials, but if aws is convoluted and confusing this might not be possible and left to the user's own debugging (gross and unfortunate).

[tbugfinder]

Well, still think there isn't an issue on AWS side.

The docker specification does not define the authentication scheme, it's just refering to RFC. Sregistry cli only implements the bearer scheme whereas AWS is using basic authentication.

[vsoch]

Makes sense :) so what is the working curl example for the linux containers, to help me fix up sregistry?

[tbugfinder]

Start with uploading an image to your registry docker-push-ecr-image.html - also see the AWS page after creating a repo.

$(aws ecr get-login.....)
docker pull busybox:latest
docker images
docker tag <local image id> <accountid>.dkr.ecr.<region>.amazonaws.com/myrepo:busyboxlatest
docker push <accountid>.dkr.ecr.<region>.amazonaws.com/myrepo:busyboxlatest
TOKEN=$(aws ecr get-authorization-token --output text --query authorizationData[].authorizationToken --region <region>)
echo $TOKEN
curl -i -H "Authorization: Basic $TOKEN" https://<accountid>.dkr.ecr.<region>.amazonaws.com/v2/myrepo/tags/list
curl -H "Authorization: Basic $TOKEN" https://<accountid>.dkr.ecr.<region>.amazonaws.com/v2/myrepo/tags/list | python -mjson.tool

[tbugfinder]

I was also trying to use sregistry --debug however there isn't any debug output. Is this another bug?

[vsoch]

I finally was able to push an image. It took over an hour to figure out this convoluted workflow, gross. The --debug isn't a bug because I (by default) set the message level to debug for the client, and it also adds --debug to singularity. So it's probably most relevant if you are interacting with singularity, which we aren't doing here (at least not yet).

[tbugfinder]

As a user I expect that --debug outputs verbose information independent of the step the program is executing. Anyway, sounds like you are now enabled to work on the ECR authentication. For me reading sregistry documentation and steps for a private registry is worse than applying well documented AWS commands. :-)

[vsoch]

Wow, your kindness makes me want to help you that must faster. Not.

[tbugfinder]

I'm just asking for having both point of views. You complained about AWS documentation which is for me very detailed and a step-by-step guide. I asked for sregistry improvements which you think is very obvious.

[vsoch]

Well I'd also point out the number of maintainers for aws is huge and here is N=1. Given that I am an open source software engineer and I do this in my free time, and that I care a lot about my users, the minimum that I ask is that you show some kindness and respect. That is also the culture of Open Source and the one that I would like to maintain on the boards that I manage.

[tbugfinder]

Please accept my honest excuse if my words above were offending. This wasn't the intention.

[vsoch]

I worked on this all day for you, and it feels terrible because you were mean. You can restore my spirit to help if you can figure out what is going on with the PR - https://github.com/singularityhub/sregistry-cli/pull/146

Basically the image / permissions work, and I get an amazon downloadURL for the layer. It returns (something) that I can extract into the singularity image, but when I build the image and run it, there is no /bin/sh in the container. So something is awry. See if you can figure it out. Thank you!

[vsoch]

Here is what you need - that should be the image and the one layer that made it. The archive looks sound, so likely it's a bug with the extraction into the sandbox. The biggest issue is that there is no metadata I can get from the registry (it's version 2 manifest) so I didn't add a runscript. That in itself might be the bug. Please take a look and let me know your thoughts, thanks.

[tbugfinder]

Running the base busybox image using docker and shell /bin/sh finishes successfully. Using singularity mount I can't find/bin.

[vsoch]

Fixed with https://github.com/singularityhub/sregistry-cli/pull/146

[tbugfinder]

thanks, for all of your efforts.

I just did another test using an image which does not have ':latest' tag but anything different. In such case the client fails to download the image. Could you verify on your end?

[vsoch]

Sure, provide me with complete documentation for pushing the image to the registry, then the calls that you make. I need to reproduce your case first, and you've given me nothing.

[tbugfinder]

# $(aws ecr get-login --no-include-email)

# docker pull ubuntu:latest
latest: Pulling from library/ubuntu

# docker pull ubuntu:latest
latest: Pulling from library/ubuntu
124c757242f8: Pull complete
9d866f8bde2a: Pull complete
fa3f2f277e67: Pull complete
398d32b153e8: Pull complete
afde35469481: Pull complete
Digest: sha256:de774a3145f7ca4f0bd144c7d4ffb2931e06634f11529653b23eba85aef8e378
Status: Downloaded newer image for ubuntu:latest
#
# docker images  | grep -i ubuntu
ubuntu                                                               latest              cd6d8154f1e1        3 weeks ago         84.11 MB
#
#
# docker tag cd6d8154f1e1 '<aws account id>.dkr.ecr.<region>.amazonaws.com/myrepo:loveaws'
# docker images  | grep -i ubuntu
ubuntu                                                               latest              cd6d8154f1e1        3 weeks ago         84.11 MB
# docker images  | grep -i love
<aws account id>.dkr.ecr.<region>.amazonaws.com/myrepo                  loveaws             cd6d8154f1e1        3 weeks ago         84.11 MB
#
#
# docker push '<aws account id>.dkr.ecr.<region>.amazonaws.com/myrepo:loveaws'
The push refers to a repository [<aws account id>.dkr.ecr.<region>.amazonaws.com/myrepo]
8d7ea83e3c62: Pushed
6a061ee02432: Pushed
f73b2816c52a: Pushed
6267b420796f: Pushed
a30b835850bf: Pushed
loveaws: digest: sha256:bb7177a4fcc6b0a8883e4fbcf5dd90e9c09a8898e0a9b258f211f654360b6cfc size: 1357
#
#
#
# sregistry  pull --name /tmp/aws-is-lovely.simg --no-cache aws://myrepo:loveaws
[client|aws] [database|sqlite:////root/.singularity/sregistry.db]
Traceback (most recent call last):
  File "/opt/anaconda2/envs/py36/bin/sregistry", line 11, in <module>
    load_entry_point('sregistry==0.0.95', 'console_scripts', 'sregistry')()
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/sregistry-0.0.95-py3.6.egg/sregistry/client/__init__.py", line 378, in main
    subparser=subparsers[args.command])
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/sregistry-0.0.95-py3.6.egg/sregistry/client/pull.py", line 51, in main
    save=do_save)
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/sregistry-0.0.95-py3.6.egg/sregistry/main/aws/pull.py", line 72, in pull
    kwargs=kwargs)
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/sregistry-0.0.95-py3.6.egg/sregistry/main/aws/pull.py", line 125, in _pull
    layers, url = self._download_layers(names['url'], digest)
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/sregistry-0.0.95-py3.6.egg/sregistry/main/aws/api.py", line 74, in download_layers
    self._get_manifest(repo_name, digest)
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/sregistry-0.0.95-py3.6.egg/sregistry/main/aws/api.py", line 112, in get_manifest
    repo = self.aws.describe_images(repositoryName=repo_name)
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/anaconda2/envs/py36/lib/python3.6/site-packages/botocore/client.py", line 623, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.RepositoryNotFoundException: An error occurred (RepositoryNotFoundException) when calling the DescribeImages operation: The repository with name 'aws/myrepo' does not exist in the registry with id '<aws account id>'
# 
#
#
# docker pull <aws account id>.dkr.ecr.<region>.amazonaws.com/myrepo:loveaws
loveaws: Pulling from myrepo
1d34996b8168: Pull complete
0d1d4c7589af: Pull complete
98890d3828f5: Pull complete
d531dc99c47b: Pull complete
b25948c45f1c: Pull complete
Digest: sha256:bb7177a4fcc6b0a8883e4fbcf5dd90e9c09a8898e0a9b258f211f654360b6cfc
Status: Downloaded newer image for <aws account id>.dkr.ecr.<region>.amazonaws.com/myrepo:loveaws

[vsoch]

Okay, so the issue here is that you aren't tagging and then asking for the right format of a container name. The correct format is basically <collection>/<namespace>:<tag> and for each <collection><namespace> this needs to exist on AWS. So for example, I would need to use a repository name that I have existing (note that I had to create this in the interface before) library/busybox and then specify my container to be there, e.g.

export CONTAINER=${SREGISTRY_AWS_ID}.dkr.ecr.${SREGISTRY_AWS_ZONE}.amazonaws.com/library/busybox:loveaws
echo $CONTAINER
692517157806.dkr.ecr.us-east-1.amazonaws.com/library/busybox:loveaws

At this point you had done something like ...amazonaws.com/myrepo:loveaws which would be parsed weirdly because there is technically no collection there. Anyway, let's continue to tag and push the (correctly named) image.

$ docker tag 16508e5c265d  "${CONTAINER}"
$ docker push ${CONTAINER}
The push refers to repository [692517157806.dkr.ecr.us-east-1.amazonaws.com/library/busybox]
ec8257ff6a7a: Pushed 
7422efa72a14: Pushed 
b6a02001ba33: Pushed 
a26724645421: Pushed 
a30b835850bf: Pushed 
loveaws: digest: sha256:ac533e4ead4110211a4d67cbf44ed8b7d1aca2b8e6f15d1e8768eadaf433dd31 size: 1357

I can check the repository names that I have with the aws tool:

$ aws ecr describe-repositories
{
    "repositories": [
        {
            "repositoryArn": "arn:aws:ecr:us-east-1:692517157806:repository/library/busybox",
            "registryId": "692517157806",
            "repositoryName": "library/busybox",
            "repositoryUri": "692517157806.dkr.ecr.us-east-1.amazonaws.com/library/busybox",
            "createdAt": 1537883304.0
        },
        {
            "repositoryArn": "arn:aws:ecr:us-east-1:692517157806:repository/library",
            "registryId": "692517157806",
            "repositoryName": "library",
            "repositoryUri": "692517157806.dkr.ecr.us-east-1.amazonaws.com/library",
            "createdAt": 1537883132.0
        }
    ]
}

Now I can look closer at the repository I pushed to in order to see tags

$ aws ecr list-images --repository-name library/busybox
{
    "imageIds": [
        {
            "imageDigest": "sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd",
            "imageTag": "latest"
        },
        {
            "imageDigest": "sha256:ac533e4ead4110211a4d67cbf44ed8b7d1aca2b8e6f15d1e8768eadaf433dd31",
            "imageTag": "loveaws"
        }
    ]
}

And here is how to correctly pull:

$ sregistry  pull --name /tmp/aws-is-lovely.simg --no-cache aws://library/busybox:loveaws
[client|aws] [database|sqlite:////home/vanessa/.singularity/sregistry.db]
Exploding /usr/local/libexec/singularity/bootstrap-scripts/environment.tar
Exploding /home/vanessa/.singularity/docker/sha256:124c757242f88002a858c23fc79f8262f9587fa30fd92507e586ad074afb42b6.tar.gz
Exploding /home/vanessa/.singularity/docker/sha256:2ebc019eb4e2bbd192e61bce91038048924216d72dfe6ac3255322caaeb70144.tar.gz
Exploding /home/vanessa/.singularity/docker/sha256:dac0825f7ffbea2ddf119026b0d6c4c453dfa38edb5c2abbe59bdba6ffdb3b9f.tar.gz
Exploding /home/vanessa/.singularity/docker/sha256:82b0bb65d1bfb978cd646dcd653164f74b7e8656aa1b3a87657b8d244b56d324.tar.gz
Exploding /home/vanessa/.singularity/docker/sha256:ef3b655c7f887451d42e45a04c2d13478171e4d451419bc8cf76c635f509532f.tar.gz
Building image from sandbox: /tmp/tmpysi4y6xp
Building Singularity image...
Singularity container built: /tmp/aws-is-lovely.simg
Cleaning up...
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Success! /tmp/aws-is-lovely.simg

Done.

[tbugfinder]

Understood and verified. Indeed, I had success for an image of <collection>/<namespace>:<sometag>.

According to ECR and the docker CLI a collection doesn't look mandatory. Could you try on your end also just by using aws://newimage:sometag ?

[vsoch]

I'm not conforming to what aws or docker does, the idea of a collection and then (some) namespace is the simple setup that Singularity (and Singularity Registry / Hub) are using, so this is how I've set it up.