Open overbit opened 4 years ago
I ran into the same problem. Reverting to Docker Desktop 2.3.0.5 made it go away, so I think this must be a new problem with 2.4.0.0.
Thanks for the report. A quick look around shows a StackOverflow post and the same error with the Azure CLI with this scenario, and a (potentially insecure) solution:
Looking at the post from the Azure CLI, it was addressed on the server side by limiting the token length. I'll pass this information on to the ECR team for review, as I don't think it's something that the CLI can address directly.
I have the same issue, also using AWS SSO
Workaround attempts:
As it stands, I'm still unable to push to ecr
I just had a call with AWS support. As far as I understand it, when you run aws ecr get-login
, you're requesting a string authentication token from AWS (IAM under the hood). There's no limit on the length of this string, but it's typically shorter than 2500 characters.
When you run docker login...
it uses Windows Credentials Manager, which doesn't support secrets of greater than 2,500 characters (see AWS support thread here: https://forums.aws.amazon.com/thread.jspa?threadID=324928). The AWS ECR guys blame AWS IAM and AWS IAM blames Windows.
With that said, we found a workaround, by using amazon-ecr-credential-manager, which seems to override Windows Credentials Manager under the hood. Follow the instructions here https://github.com/awslabs/amazon-ecr-credential-helper. This has the added benefit of automating the aws ecr get-login
step, so you can go straight from you docker tag
step to docker push
step.
@maurera Thanks for the help. Unfortunately, it doesn't seem that amazon-ecr-credential-manager supports AWS SSO at the moment. https://github.com/awslabs/amazon-ecr-credential-helper/issues/229
Hi @maurera, thanks for passing along that information. I'm going to mark this as a potential update for our documentation troubleshooting section.
Sorry that the solution provided doesn't work for you, @overbit. You can add in a comment on the issue you linked.
FYI - the ECR Credential helper initially helped in my case, but now I get the following error after running docker push
:
denied: Your authorization token has expired. Reauthenticate and try again
Hi @maurera, sorry to hear that. This seems like an issue with the credential helper, so I would suggest following up in their repository.
Note - I tried two workarounds.
First (didn't work). I launched a WSL2 Ubuntu Linux instance and tried aws ecr get-login-password
from there. This didn't work (it seems like it's still connecting with the docker engine running in Windows, which connects to Windows Credential Manager.)
Second (works). I spun up an EC2 instance, copied over my project, and docker push
d from there.
This seems like a major issue. People who develop in Windows can't push Docker containers to AWS right now.
@maurera I had the same issue with WSL. Fixed it by deleting the docker.config
from Windows and restarting everything. docker-login
started using the fallback of storing creds in the docker.config on WSL. Go figure.
I did, at one point, have docker push working in WSL with the ECR credentials plugin. Then it started failing again. I haven't had the time to really hammer out the problem, but I will. In the meantime, I hope that helps you get the work-around working in WSL.
Hi @maurera and @paulriley, thanks for the further details and feedback on the impact of this issue. I agree that it is a major issue impacting Windows developers. I have let the ECR team know about it. I would recommend opening (or re-opening) any issue you have with AWS Support to escalate it further.
The get-login-password
is a wrapper around the boto3 ECR client get_authorization_token
command, which is a direct call to the ECR API GetAuthorizationToken
action:
The length of the token you get back is dependent on the type of IAM principal account. Unfortunately there isn't anything that the AWS CLI can do to change this behavior.
@kdaily is it worth to open a new issue about the length of the token received by ECR API? If yes, what will be the best repository to open the issue against?
This seems to only happen with accounts that have AWS SSO enabled.
The workaround we've found from StackOverflow is to open the %userprofile%/.docker/config.json file, and remove the credStore
and credsStore
properties.
The problem is it's only temporary. Each day the engineers need to run aws sso login
, and each day they need to open the above file and remove those values before calling aws ecr get-login-password | docker login --username AWS --password-stdin <registry>
I can confirm that aws ecr get-login-password
returns a string greater than 2,500 characters when AWS SSO is enabled.
SSO, maybe. But not specifically AWS.
We're using Azure SSO to access AWS and seeing the same problem.
It's possible the long keys are for accounts that have assumed roles. But, as I see it, the key length is not the problem, it's wincred's inability to handle such keys.
I've created an issue over at Docker to indicate the bug with the Windows credential helpers. https://github.com/docker/docker-credential-helpers/issues/190
Issue :
Error saving credentials: error storing credentials - err: exit status 1, out: The stub received bad data.
exit status 1
Analysis:
saml2aws exec -a devstage "aws ecr get-login-password --region eu-central-1"
The above command returns a big password. Wincred seems to work with passwords of size 2500 characters or less.But the above command returns password with 2500 characters or more, in my case it was 2580 characters.So AWS ECR login fails.The docker desktop version I have in my windows10 is 2.5.0.0 .To solve this problem we have to do two things.
Solution :
1) Rename docker-credential-wincred.exe in C:\Program Files\Docker\Docker\resources\bin\ to say docker-credential-wincred123.exe
2) Go to C:\Users\
Note : config.json will be created after the first successful login.So if you don't find config.json in C:\Users\\<user.name>\\.docker, do a normal docker login with your docker account and password and the file will be created. Then go and do the point no 2 above and logon to ECR.
Now we will be able to logon to AWS ECR successfully in a command prompt with a warning.
Hope this helps !!.
I've created a workaround project that avoids the Windows Credential Manager, as it's completely blocking for us.
Use with caution
https://github.com/dougrday/docker-credential-plaintext
In our case, this workaround is fine as we're using SSO, and our tokens are short-lived (4 hours). If Docker Desktop resolves the issue with Windows Credential Manager, we'll stop using the workaround.
FWIW, I (re)moved the ~/.docker/config.json
and then the login succeeds as described in the docs. I'm running this in WSL2.
aws-cli/2.1.3 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.20
Client:
Version: 19.03.8
API version: 1.40
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Oct 14 19:43:43 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 20.10.0-rc1
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: 131bf7e
Built: Tue Nov 17 22:52:57 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.4.1
GitCommit: c623d1b36f09f8ef6536a057bd658b3aa8632828
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
@overbit commented on Oct 16
Thanks for the help. Unfortunately, it doesn't seem that amazon-ecr-credential-manager supports AWS SSO at the moment.
@maurera I've posted a workaround that might help in some SSO situations: https://github.com/awslabs/amazon-ecr-credential-helper/issues/229#issuecomment-738895977
I've create my own version of the docker credential store to overcome this problem. It is not the best solution since it will request the credentials every time docker needs to interact with the AWS ECR registry but at least there are not security concerns around temporarily storing credentials on the local machine.
The only requirement is to have aws cli v2 installed. For more info: https://github.com/overbit/docker-credential-aws-sso-ecr
Unfortunately no workaround works
I was bitten by this using Windows 10 + WSL2 + awscliv2 and @mklinke suggestion worked for me. Just make sure you're removing ~/.docker/config.json
inside the WSL2 VM, not in Windows.
Continues to be an issue unfortunately, really bad experience trying to do anything with aws ecr and docker
It is now over a year later and you still have not fixed this? Are you going to ? Surely seems like an easy fix...
password = generate_password(...)
if(password.length > max_password_length)
password = password.substring(0, max_password_length)
I think a fix will need to be an adapter/update to the Wincred logic that talks to the OS Wincred API (that has the limit) to know it needs to make multiple entries as a series of parts to join back together (I had a quick look), while keeping backwards comparability
Seems to me if there is a restriction on the length of the password on windows it would be easy enough for aws to restrict the length of the passwords they generate for windows... I don't really mind what the solution is, but expecting their customers to waste hours on something that should just work is very disappointing.
I have created a request on the container roadmap: https://github.com/aws/containers-roadmap/issues/1589
Please react to it with 👍 so that it may be appropriately prioritized for consideration.
For what it's worth, think this is a permissions issue in the terminal.
Created a docker group and assigned the docker.sock to the group. Added the docker command to the group and then assigned w permission to the socket for the group. Not a fan of not using sudo
, but seems to be the only way around it for the short term.
this worked with sudo but has issues with downstream things I am doing
aws ecr get-login-password --region <specific-region> | sudo docker login --username AWS --password-stdin <myaccount>.dkr.ecr. <specific-region>.amazonaws.com
downstream code doesn't use sudo and falls over.
Confirm by changing [ ] to [x] below to ensure that it's a bug:
Describe the bug Authenticate Docker to an Amazon ECR registry with get-login-password fails with the following error:
Error saving credentials: error storing credentials - err: exit status 1, out: 'The stub received bad data.'
SDK version number aws-cli/2.0.56 Python/3.7.7 Windows/10 exe/AMD64
Platform/OS/Hardware/Device Windows 10 1909 Docker Desktop 2.4.0
To Reproduce (observed behavior)
aws ecr get-login-password --region <specific-region> | docker login --username AWS --password-stdin <myaccount>.dkr.ecr. <specific-region>.amazonaws.com
Expected behavior Login should be successful
Additional context I'm currently using an aws profile configured with AWS SSO
aws ecr get-login-password --region <specific-region>
command alone succeed returning a token. This token might be too big for docker login to accept?