Doesn't work with AWS SSO

[otaviomedeirossb]

I'm trying to make it work with AWS SSO but i'm getting NoCredentialProviders.

I used aws configure sso to create a profile on my local machine and i can manually docker login and pull images from ECR like this:

aws ecr get-login-password --region us-west-2 --profile my-profile | docker login --username AWS --password-stdin my-account-id.dkr.ecr.us-west-2.amazonaws.com

docker pull account-id.dkr.ecr.us-west-2.amazonaws.com/my-repo:latest

My ~/.aws/config looks something like:

[profile my-profile]
sso_start_url = https://my-company.awsapps.com/start
sso_region = us-east-1
sso_account_id = my-account-id
sso_role_name = MyRoleName
region = us-west-2
output = json

Nothing was added to ~/.aws/credentials.

I logged out docker, installed docker-credential-helper-ecr and set:

~/.docker/config.json

{
   "credsStore": "ecr-login"
}

and:

export AWS_PROFILE=my-profile
export AWS_REGION=us-west-2

When i:

docker pull account-id.dkr.ecr.us-west-2.amazonaws.com/repo:latest

I get:

no basic auth credentials

It works fine on another AWS account which is not SSO and i have my access keys set to ~/.aws/credentials.

Does it work with AWS SSO somehow?

[FernandoMiguel]

i assume you did aws sso login ?

[otaviomedeirossb]

@FernandoMiguel yes i did and i can pull the images if i manually login to docker.

[awilkins]

Not everything can read the credential store that SSO uses, which is a bunch of JSON files in ~/.aws/sso/cache, but they contain the same stuff you'd get from any other sts:AssumeRole - access key id, secure access key, and session token - albeit encoded as a JWT.

Maybe try this small util I wrote that does an SSO login and copies the credentials into your "normal" ~/.aws/credentials file.

https://github.com/awilkins/aws-sso-refresh/

[lachlancooper]

The solution I use with other applications that do not yet natively support AWS SSO is an external credential_process, using a tool like https://github.com/benkehoe/aws-sso-credential-process

However, this doesn't currently work with amazon-ecr-credential-helper. In the logs I get an error like the following:

... error="ecr: Failed to get authorization token: ProcessProviderParseError: parse failed of credential_process output: <JSON REDACTED>\ncaused by: unexpected end of JSON input"

This is because the output from the credential process is longer than 1024 characters, but the max buffer size was only increased in aws-sdk-go v1.25.42.

Until proper SSO support is added here, it would be great to bump the SDK version to allow use of credential_process as a workaround.

[overbit]

This issue is becoming more important with the release of the latest version of docker v2.4.0. There seems to be a limit of 2500 characters on the auth token that makes the aws ecr get-login-password method not working (check here https://github.com/aws/aws-cli/issues/5636).

[rhertogh]

This issue is becoming more important with the release of the latest version of docker v2.4.0. There seems to be a limit of 2500 characters on the auth token that makes the aws ecr get-login-password method not working (check here aws/aws-cli#5636).

It's not a solution in all situations but for those looking for a workaround for trying to push from a local computer the following might work:

• Navigate to https://your-company.awsapps.com/start
• Under the desired account select "Command line or programmatic access"
• Copy the AWS environment vars for the desired operating system
• Paste them in a console
• Amazon-ecr-credential-helper will now work (at least it does for me 😁)

[dougrday]

For those that need to support many teams with this, I've created a temporary workaround credential helper. https://github.com/dougrday/docker-credential-plaintext

It's working with our teams with AWS SSO, but it's definitely not the "secure" solution. Our tokens only live for 4 hours though, so the security window is fairly small.

I'd definitely prefer that other credential helpers step up and resolve the issue in a more secure fashion.

[guilhermeheinen]

@dougrday . Your solution is the only one that worked for me. I understand the risk of exposing the password for a short period of time. That said, it is the only workaround until other credential helpers solve this issue. Thanks!

[ericvilla]

I @otaviomedeirossb, as already stated, AWS CLI v2 will write AWS SSO temporary credentials under ~/.aws/cli/cache folder, which in many cases ends up to a "credentials not found" issue. I'm currently working on an open source project that addresses this problem too. Indeed, it provides support to AWS SSO and lists you all the IAM Roles that your AWS SSO User can access. By clicking a session card associated with a IAM Role, Leapp will generate temporary credentials to access your AWS Organization's Account, inside ~/.aws/credentials file. Let me know if it makes sense to you and if you have questions about it.

[TylerLubeck]

I've also taken a stab at this, and ended up with something that doesn't rely on node and doesn't store temp credentials anywhere*. It's working well for me, and hopefully it's of some value to the rest of you as well.

https://github.com/TylerLubeck/docker-credentials-aws-ecr-sso

*Not to say that these things are bad, they just don't fit for my use case

[flyinprogrammer]

The solution I use with other applications that do not yet natively support AWS SSO is an external credential_process, using a tool like https://github.com/benkehoe/aws-sso-credential-process

However, this doesn't currently work with amazon-ecr-credential-helper. In the logs I get an error like the following:

... error="ecr: Failed to get authorization token: ProcessProviderParseError: parse failed of credential_process output: <JSON REDACTED>\ncaused by: unexpected end of JSON input"

This is because the output from the credential process is longer than 1024 characters, but the max buffer size was only increased in aws-sdk-go v1.25.42.

Until proper SSO support is added here, it would be great to bump the SDK version to allow use of credential_process as a workaround.

I can confirm this particular issue with credential_process not working with was fixed with this PR: https://github.com/awslabs/amazon-ecr-credential-helper/pull/240 - however a version of amazon-ecr-credential-helper hasn't been released with this update.

As for native support of SSO I imagine this is stalled until something like this: https://github.com/aws/aws-sdk-go/pull/3610 is merged to the SDK, then it will Just Work™ with another SDK upgrade because it seems it'll be part of the default profile credential parser.

Also, sidebar - it seems that Docker Desktop for Mac is overwriting /usr/local/bin/docker-credential-ecr-login when the application is started, which is seemingly terrible and and they should feel bad for doing that. So that's also a mess 🤸

[masteinhauser]

Now that https://github.com/aws/aws-sdk-go/pull/3610 has merged and is released as part of aws-sdk-go v1.37.0+, when will this dependency be upgraded in the project? Currently it is v1.36.0, and while there are significant changes the aws-sdk-go CHANGELOG.md doesn't appear to immediately call out risks for this project.

This is one of the final tools we rely on to formally adopt the native AWS SSO credential protocol for us to deprecate our custom shim scripts.

[stijndehaes]

I was wondering if a release is planned in the next coming weeks :) This is also one of the last tools I need for AWS SSO adoption. For now I guess I will build it from source, but a release would be better!

[tristanpemble]

For anyone wandering in from Google, I can confirm that #267 fixes this for me after building from source.

[shavo007]

hi @tristanpemble is there a new release of this with that fix or you have to build from source?

[awilkins]

Last release, Feb 15th, date of the MR that fixes this, Feb 22nd.

Yup, you still have to build from source.

[shavo007]

thanks @awilkins sad

anyone here able to kick off a release and get it on brew?

[jobirobi]

Seconding (Nth-ing?) the request for a homebrew release with this change included 🙏

[tristanpemble]

we use Nix here. in the off chance that you are also using Nix, here is the overlay that we use:

self: super:
{
  amazon-ecr-credential-helper = super.amazon-ecr-credential-helper.overrideAttrs(old: {
    src = super.fetchFromGitHub {
      owner = "awslabs";
      repo = "amazon-ecr-credential-helper";
      rev = "efd1603705e91c31bf28cbbc5dba8c3e09c63100";
      sha256 = "sha256:0d2nc4rl0pl3mspvavxixa6lsjnnx5g05srpw1f68wqbj0vb7kd1";
    };
  });
}

[kyledecot]

This is still a problem for Homebrew users as the current formula is pinned at 0.5.0 (b19192b6522b2da02d14ec394c331f3b1a70efe2).

[ghost]

It looks like the Homebrew formula is updated now. In my case, I had to force-link it after installing, to overwrite the 0.5.0 version installed with Docker Desktop. I also had to clean up a bunch of pre-SSO config, but the most relevant things were:

• Deleting old ECR cache: rm ~/.ecr/cache.json
• Deleting old AWS CLI credentials: rm ~/.aws/credentials (not 100% sure if this is required, but it's not needed/used anymore with SSO)
• Re-logging to establish new tokens in the expected places: aws sso login

After all of the above, I was able to painlessly pull images from ECR again.

[gautam-nutalapati]

I am using amazon-ecr-credential-helper for first time, I installed 0.6.0 version via brew. When I pull from private ECR repo, I see Error response from daemon: Head "https://***.dkr.ecr.us-east-1.amazonaws.com/v2/prisidio/service-base-image/manifests/latest": no basic auth credentials

What am I missing? 🤔 I use AWS SSO. Exporting correct profile, and sos sso login as well.

My dockerconfig is:

{
        "credsStore": "ecr-login",
        "credHelpers": {
          "public.ecr.aws": "ecr-login",
          "XXX.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        }
}

brew info docker-credential-helper-ecr                                                                                                                                   main
==> docker-credential-helper-ecr: stable 0.6.0 (bottled)
Docker Credential Helper for Amazon ECR
https://github.com/awslabs/amazon-ecr-credential-helper
/usr/local/Cellar/docker-credential-helper-ecr/0.6.0 (7 files, 8.6MB)
  Poured from bottle on 2022-09-16 at 09:59:58
...

[ejhayes]

@gautam-nutalapati make sure that you are setting environment variable AWS_SDK_LOAD_CONFIG=1 and that your AWS profile has permissions to ECR. Everything else you are doing looks correct.

[gautam-nutalapati]

I tried this, but unfortunately same thing Error response from daemon: Head "https://651627600016.dkr.ecr.us-east-1.amazonaws.com/v2/prisidio/service-base-image/manifests/latest": no basic auth credentials Same dockerconfig as above and my profile is admin profile, I validated list-images to ensure SSO profile is setup correctly, which is.

[kzys]

Let us take a look this week.

[matuszeman]

@gautam-nutalapati I had the same problem as you reported. I fixed it by setting AWS profile via environment variable.

AWS_PROFILE=<your-profile> docker pull <ecr-id>.dkr.ecr.eu-central-1.amazonaws.com/<repo>

[natenho]

Same problem here, setting AWS_PROFILE= doesn't work aswell. I tried many different combinations and the credential helper did not work. no basic auth credentials

[matuszeman]

Does your IAM user have ecr:GetAuthorizationToken permission allowed?

[natenho]

@matuszeman yes, I have the permsision GetAuhtorizationToken permission. I'm also able to perform pulls authenticating via aws ecr get-login-password | docker login, but not via credential helper =/

I noticed a log file inside .ecr folder with this line that might help:

time="2022-10-04T00:25:03-03:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

[chaoqunya]

This command works

❯ aws ecr get-login-password --region ap-east-1 --profile myprofile | docker login --username AWS --password-stdin XXX.dkr.ecr.ap-east-1.amazonaws.com
❯ docker push XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
The push refers to repository [XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo]
370e6XXXXXXX: Pushed 

latest: digest: sha256:XXXXXXXXXX size: 1987

But this don't, is there anyone has similar problem?

❯ AWS_PROFILE=myprofile docker push XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
The push refers to repository [XXX.dkr.ecr.ap-east-1.amazonaws.com/my-ecr-repo:latest
370e6XXXXXXX: Preparing 
no basic auth credentials
❯ cat ~/.ecr/log/ecr-login.log
time="2023-02-02T10:48:11+08:00" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="the SSO session has expired or is invalid: open /Users/me/.aws/sso/cache/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json: no such file or directory"
time="2023-02-02T10:48:11+08:00" level=debug msg="Retrieving credentials" region=ap-east-1 registry=XXX serverURL=XXX.dkr.ecr.ap-east-1.amazonaws.com service=ecr
time="2023-02-02T10:48:11+08:00" level=debug msg="Calling ECR.GetAuthorizationToken" registry=XXX
time="2023-02-02T10:48:11+08:00" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: operation error ECR: GetAuthorizationToken, failed to sign request: failed to retrieve credentials: the SSO session has expired or is invalid: open /Users/me/.aws/sso/cache/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json: no such file or directory"

[rchennault-exelixis]

Yes. I can reproduce.

[vtatarin]

FYI, SSO-related issues seem to be gone in a new binary version (077b4a9), which can be built manually from the sources. Not sure why new releases are not posted for such a long time already

[gondalez]

FYI, SSO-related issues seem to be gone in a new binary version (077b4a9), which can be built manually from the sources. Not sure why new releases are not posted for such a long time already

Thanks for the heads-up @vtatarin. I've had a few issues with SSO bugs in my team.

@samuelkarp @austinvazquez I noticed you have changed CHANGELOG.md in the past... any insights as to why there are master commits but no releases past 0.6.0 for ?

I'd love to see the newer version with bugfixes released. I assume that would allow it to make its way into a future Docker Desktop release as well for ultimate 😌

This is what I see locally; my credentials helper is packaged with with docker and pinned at 0.6.0:

» which docker-credential-ecr-login
/usr/local/bin/docker-credential-ecr-login

» ls -lah /usr/local/bin/docker-credential-ecr-login

lrwxr-xr-x  1 root  wheel    75B  1 Dec  2021 /usr/local/bin/docker-credential-ecr-login -> /Applications/Docker.app/Contents/Resources/bin/docker-credential-ecr-login

» docker-credential-ecr-login -v
amazon-ecr-credential-helper
Version:    0.6.0
Git commit: 69c85dc22db6511932bbf119e1a0cc5c90c69a7f

[samuelkarp]

@samuelkarp I noticed you have changed CHANGELOG.md in the past...

@gondalez I no longer work at Amazon.

[adampl]

Thanks! I've wasted half a day on trying to fix this, until I ended up here...

Installing from source did the job:

go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest

[austinvazquez]

@gondalez thanks for the interest in the project and the feedback. As others have noted building from main will help the immediate pain. That said let me bring this up with the team to see about an updated release with these fixes.

[austinvazquez]

Special thanks to @fangn2 who worked to push this through. v0.7.0 is available with fix for the issue.

[zchenyu]

Ran into the same issue. Solved by installing from source per the README

go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest

and also explicitly setting AWS_PROFILE.

Is there a way for the helper to use the default profile if AWS_PROFILE is not specified ? It's a bit annoying to manually specify the profile.

[paminhoff]

Works for one day once creds expire it breaks. Guess no longer supported properly by AWS. Back to unsecure ways of doing things until something stable comes along. Tried everything posted here including the profile and updating from go, rming files under .ecr and .aws/credentials

[gopipal]

@paminhoff I wonder if you need to do a docker logout "<aws-user-id>.dkr.ecr.<aws-cli-region>.amazonaws.com" every time the creds expire after a day/12 hrs(since docker can cache creds) before you login using aws sso login.

I am using the latest aws credentials helper and it works with aws sso

go install github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest
sudo cp $GOPATH/bin /usr/bin/.

This is my ~/.docker/config.json

{
    "credHelpers": {
        "<aws-user-id>.dkr.ecr.<aws-cli-region>.amazonaws.com": "ecr-login"
    }
}