awslabs / amazon-ecr-credential-helper

Automatically gets credentials for Amazon ECR on docker push/docker pull
Apache License 2.0
2.5k stars 339 forks source link

docker-credential-ecr-login breaks OSX network connection forces reboot #381

Open andrericardo opened 2 years ago

andrericardo commented 2 years ago

docker-credential-ecr-login breaks OSX network connection, the only way I found so far to recover is to reboot the laptop.

All hardware network devices disappear eg networksetup -listallhardwareports no longer works, also have a screenshot below from Network Preferences.

The issue is so weird and hard to believe that made three videos and attached.

$ docker-credential-ecr-login -v
amazon-ecr-credential-helper
Version:    0.6.0
Git commit: 69c85dc22db6511932bbf119e1a0cc5c90c69a7f

$ docker-credential-ecr-login version
0.6.3
$ docker --version
Docker version 20.10.16, build aa7e414

$ cat ~/.docker/config.json
{
  "auths": {},
  "credHelpers": {
    "087665217675.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login"
  },
  "credsStore": "desktop",
  "experimental": "disabled",
  "stackOrchestrator": "swarm"
}
$ cat ~/.aws/credentials
[default]
    region = eu-west-1

Steps to replicate, setup AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

aws-vault exec unstable-read-only -- env | grep AWS
export AWS_ACCESS_KEY_ID=(the keys from above)
export AWS_SECRET_ACCESS_KEY=

Then run docker-credential-ecr-login list

The command docker-credential-ecr-login get <<< 087665217675.dkr.ecr.eu-west-1.amazonaws.com also causes this issue.

Is there a cached file somewhere that could be causing this? I've removed and reinstalled docker-credential-ecr-login already.

Any help just on how to kill docker-credential-ecr-login and/or recover network connection would be useful.

Screenshot 2022-08-25 at 15 57 54

https://user-images.githubusercontent.com/171730/186751151-495ff5e7-2c78-479a-abc6-c3221e02237d.mov

To prove it was not just one time fluke

https://user-images.githubusercontent.com/171730/186757490-c1ea2214-4599-4557-b3ab-81a4be51c253.mov

Third time with clean ~/.ecr and logging on ~/.ecr/log/ecr-login.log

https://user-images.githubusercontent.com/171730/186760164-823d2270-6801-4e67-935a-ab5cb2ea5c5b.mov

ecr-login.log

andrericardo commented 2 years ago

Alternative to docker-credential-helper-ecr as per https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html#registry-auth-token

Also used this to confirm the issue is not in aws, aws-vault or docker pull.

First you need aws to work, use

$ aws-vault exec [your profile] -- env | grep AWS_

Pass the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to

$ aws configure

If using SSO you need to login, your login method probably will be different but for me it is

$ aws sso login --profile [some login profile]

Then login into Docker with the aws credentials

$ aws ecr get-login-password --profile [your profile] --region [region] | docker login -u AWS --password-stdin https://[aws_account_id].dkr.ecr.[region].amazonaws.com/
Login Succeeded

$ docker pull ...
kzys commented 2 years ago

This is really surprising to be honest. The helper shouldn't have this level of power...

frison commented 2 years ago

@andrericardo Yeah, we had to adopt a similar workaround and did not uncover a root-cause for this. Any chance you've made progress?

andrericardo commented 2 years ago

I had to upgrade my OSX version ProductVersion: 12.6 BuildVersion: 21G115 and now have trouble getting the list command to show the relevant credHelpers...

$ cat ~/.docker/config.json
{
  "auths": {},
  "credHelpers": {
    "087665217675.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login"
  },
  "credsStore": "desktop",
  "experimental": "disabled",
  "stackOrchestrator": "swarm"
}

$ docker-credential-ecr-login list
{}

Brew installed

$ /usr/local/bin/docker-credential-ecr-login list
{}

Docker.app installed one

$ /Applications/Docker.app/Contents/Resources/bin/docker-credential-ecr-login list
{}

Also I've cloned this repository and now can run ecr-login/cli/docker-credential-ecr-login/main.go in debug mode on VS Code. I don't have prior experience in Go but can breakpoint in some file and paste here the variables if it's any help...

Got the same {} in debug mode.

image