Cannot pull images from AWS ECR (login does not seem to work properly)

[mobileben]

I'm currently using docker-py 3.7.0 on an Ubuntu VM running Docker version 17.09.0-ce.

I'm having difficulty in what appears to be properly logging into docker. I've tried to get the AWS ECR credentials one of two ways: via boto3 and calling a subprocess for aws ecr get-login.

What happens is that when I try and pull an image, I get the dreaded

repository does not exist or may require 'docker login'

message.

I invoke this script with sudo (eg. sudo ./myscript.py). If, prior to running the script, I run

aws ecr get-login --no-include-email --region us-west2

and then run the results with sudo, the script will properly run.

I've tried variations and even used reauth during login. When I do that, I get the response

http://localhost:None "POST /v1.35/auth HTTP/1.1" 200 48 login_results {'IdentityToken': '', 'Status': 'Login Succeeded'}

I've even deleted the ~/.docker/config.json file but this doesn't help (a new file isn't even written).

here is a code snippet of what I'm doing for the login. It's a little messy right now since I've been trying permutations

    command = "aws ecr get-login --no-include-email --region us-west-2"
    p = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
    (out, err) = p.communicate()
    outstr = out.decode("utf-8")
    errstr = err.decode("utf-8")
    if p.returncode == 0:
        # Remove prefix
        outstr = outstr.lstrip('docker login ')
        parts = outstr.split(' ')
        print(parts)
        # -u
        username = parts[1].strip()
        # -p
        password = parts[3].strip()

        registry_url = parts[len(parts)-1].strip()
    else:
        print(p.returncode)
    '''
    token = ecr_client.get_authorization_token()
    username, password = base64.b64decode(token['authorizationData'][0]['authorizationToken']).decode('utf-8').split(":")
    registry_url = token['authorizationData'][0]['proxyEndpoint']
    '''
    print('username {}'.format(username))
    print('password {}'.format(password))
    print('registry_url {}'.format(registry_url))
    docker_client = docker.from_env()
    treg = registry_url + '/'
    login_results = docker_client.login(username=username, password=password, reauth=True, registry=treg)
    print('login_results {}'.format(login_results))

[mobileben]

One other thing to note here. I mentioned that if I get login and then log into docker prior to running the script it works. I noticed it also recreates the config.json file, whereas calling it from python does not. If config.json does exist, the entry is not updated when run from python.

If I run login as a subprocess, config.json does get modified. This behavior seems rather strange.

[mobileben]

I need to do a little more testing here, but if I run docker login as a subprocess. (which defeats the purpose of using docker-py), it works.

It seems that the problem comes from when config.json contains an entry for the registry with a stale auth token.

If config.json does not exist or if config.json exists but doesn't have an entry for the registry, it will work properly. Note this is independent of whether or not the reauth parameter is passed in with login.

[mobileben]

Okay, I ran some experiments the last few days. Had to deal with the 12 hour AWS ECR ticket so it took a little longer to do.

It does seem that there is an issue with docker-py.

Based on my findings, I can either use boto3 or run a sub-process calling the command line to aws ecr. However the only permutation that seems to work with the following steps.

1. use a sub-process to perform the docker login. This will result in the config.json file being updated (not sure if this has any relevance at all or not).
2. Create the docker client via docker_client = docker.from_env(). I have found doing this prior to the sub-process results in it not working properly (unless you have an already valid config.json
3. Then call docker_client.login(username=username, password=password, registry=registry_url)

Whether or not this is expected or not or if I'm doing something wrong, I don't know. This is what I've come up with as steps that work.

[mikeage]

I'm not sure if this is related to your issue, but I'm having the same problem, and I was able to resolve it by calling docker_client.login with registry equal to registry_url = token['authorizationData'][0]['proxyEndpoint'].replace("https://", ""). Note that I'm removing the leading https://

[jonapich]

the docker.login() response usually has a Status: Login Succeeded when it works, but when the user has a ~/.docker/config.json then the reply will be a totally different payload, containing the username/password (AWS and the super long b64 string use for docker login) instead of actually performing the login.

2 of my collegues had that issue, and after deleting the config.json the response was appropriate again.

[delwaterman]

Bump on this issue. I can't just delete the ~/.docker/config.json since I am using a combination of GCR and ECR and this will run on developer machines

[norton120]

Looks like this is still an issue? My workaround has been to wrap the whole thing in a context manager that clears out the config.json first, then does the work, and finally returns it to the original state. It's not pretty but it does the job. (updated to use the decorator)

from contextlib import contextmanager

    @contextmanager
    def _flush_existing_login(registry: str) -> None:
        """ handles the known bug 
            where existing stale creds cause login
            to fail.
            https://github.com/docker/docker-py/issues/2256
        """
        config = Path(Path.home() / ".docker" / "config.json") 
        original = config.read_text()
        as_json = json.loads(original)
        as_json['auths'].pop(registry, None)
        config.write_text(json.dumps(as_json))
        try:
            yield
        finally:
            config.write_text(original)

[tehsven]

I also just ran into this issue today. The workaround provided by @norton120 is working for me.

[dugarsumit]

You can also set reauth=True from here https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient.login

[durka]

This seems like a significant bug that should at least be documented! Ideally, login would throw an exception instead of silently doing nothing. The combination of removing https:// (@mikeage) and setting reauth=True (@dugarsumit) solved the issue for me. Either of those on its own didn't change the behavior at all.

[utkarsh867]

I had been struggling with this issue for a while. It seems like running a logout first works.

I did a

docker logout ***.dkr.ecr.*region*.amazonaws.com

This fixed the docker-py for me.

[anenoglyadov]

That's still an issue nowadays. Removing the ~/.docker/config.json works. Also possible way is to call "docker logout ...", but there is no python equivalent for that command in SDK. So there are no beautiful workarounds to fix that strange behaviour, which definitely should be fixes at some point, I hope.

[bruce-szalwinski]

I've done the https:// removal as well as the reauth=True but I'm still running into issues. I have a python service I'm deploying to ECS. The heart of the service is:

        docker_client = docker.from_env()
        username, password = ecr.credentials()
        registry = repo.registry_uri().replace("https://", "")
        docker_client.login(
            username=username, password=password, registry=registry, reauth=True
        )
        docker_client.images.pull(repository=repository, tag=tag)

On my local mac, I can run the above in Pycharm, works great. I can take the same script an bundle it in a container and run the container on my local machine, works great. I then push the container to ECR, and launch the container in ECS. I try and pull the image and I get the dreaded

ERROR: 404 Client Error for http+docker://localhost/v1.40/images/xxx.dkr.ecr.rrr.amazonaws.com/python:3.6.8/json: Not Found ("no such image: xxx.dkr.ecr.rrr.amazonaws.com/python:3.6.8

Ah, found the problem. I was missing ecr:GetDownloadUrlForLayer in the policy for the task.

[bmoller]

I too recently ran into this issue, but I don't think it's a docker-py bug per se. I discovered the following after digging through the docker source code (written in Go).

If you refer to the Docker API documentation the /auth endpoint actually only tests that registry credentials are valid, and optionally returns a token if the tested registry responds with one for future authentication. There is nothing about credential storage because the daemon does not, in fact, store credentials ever. Any credential storage that takes place is handled by the docker binary itself externally to the API.

When executing docker login on the command line your config.json file is checked for any configured credential store; whatever string is found there is appended to docker-credential- and executed. The Docker project maintains a repo for the most common helpers: https://github.com/docker/docker-credential-helpers

I was specifically using this library for integration with ECR as well; my process now looks like this:

1. Use boto3 ECR client to get Docker credentials
2. Validate the credentials with the docker-py function login(), if desired
3. Use the subprocess Python standard library to call docker-credential-<some configured helper> and pass the ECR credentials on stdin

That being said, I think the docker-py documentation should be updated to call out this fact, and could be patched with credential storage similar to the CLI binary. The way it's currently described implies similar functionality to docker login on the command line, which is simply not the case.

[elisiariocouto]

Sorry to hijack this, I found a bug that might be related to this. When pushing an image to ECR, if my credentials are expired, the client.images.push is not raising an error. Is this the expected behaviour?

[linuxtek-canada]

Ran into this issue as well, and can confirm @mikeage and @dugarsumit combined changes worked for me as well.