In our company, we intend to do image copy from public repository to our private ECR repository, just because we don't want to encounter network issue that may cause the image pulling to be slow/failed.
Currently our naming convention of the repository looks like: $ecr_repository_url$public_repository_name:$image_tag.
A concrete example would be, for image public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12, we'll do a replication as 112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12
So as for kubelet's image pulling, we're facing Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image 112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit: exit status 1.
The above error results to image pulling failed with 401 Unauthorized issue.
After digging the source code, I've noticed that the ecrPlugin.GetCredentials uses a keyword public.ecr.aws to determine if the given image is a public repository or not.
Thus in our case, a credential request was falsely made to publilc registry instead of private registry.
What you expected to happen:
Even with the keyword public.ecr.aws present in the image url, the ecr-credential-provider shall still be able to determin if the image is in private registry or not.
How to reproduce it (as minimally and precisely as possible):
Create a private repository and image with keyword public.ecr.aws, for example, 112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12.
Then invoke directly the ecr-credential-provider with:
echo '{"kind":"CredentialProviderRequest","apiVersion":"credentialprovider.kubelet.k8s.io/v1","image":"112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12"}' | /etc/eks/image-credential-provider/ecr-credential-provider
I0904 12:24:55.935639 30684 main.go:95] Getting creds for public registry
E0904 12:24:57.143893 30684 main.go:261] Error running credential provider plugin: UnrecognizedClientException: The security token included in the request is invalid
status code: 400, request id: 05cad298-32ce-4775-85b2-d23ce746f26e
For comparaision, use another repository and image, 112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/aws-observability/aws-for-fluent-bit:2.31.12:
cartermckinnon
commented
1 year ago
What happened:
In our company, we intend to do image copy from public repository to our private ECR repository, just because we don't want to encounter network issue that may cause the image pulling to be slow/failed.
Currently our naming convention of the repository looks like:
$ecr_repository_url$public_repository_name:$image_tag.A concrete example would be, for image
public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12, we'll do a replication as112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12So as for kubelet's image pulling, we're facing
Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image 112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit: exit status 1.The above error results to image pulling failed with 401 Unauthorized issue.
After digging the source code, I've noticed that the
ecrPlugin.GetCredentialsuses a keywordpublic.ecr.awsto determine if the given image is a public repository or not.Thus in our case, a credential request was falsely made to publilc registry instead of private registry.
What you expected to happen:
Even with the keyword
public.ecr.awspresent in the image url, theecr-credential-providershall still be able to determin if the image is in private registry or not.How to reproduce it (as minimally and precisely as possible):
Create a private repository and image with keyword
public.ecr.aws, for example,112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/public.ecr.aws/aws-observability/aws-for-fluent-bit:2.31.12.Then invoke directly the
ecr-credential-providerwith:For comparaision, use another repository and image,
112233445566.dkr.ecr.cn-northwest-1.amazonaws.com.cn/aws-observability/aws-for-fluent-bit:2.31.12:Anything else we need to know?:
Nein.
Environment:
kubectl version):Server Version: version.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.4-eks-2d98532", GitCommit:"3d90c097c72493c2f1a9dd641e4a22d24d15be68", GitTreeState:"clean", BuildDate:"2023-07-28T16:51:44Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"linux/amd64"}uname -a):Linux ip-172-24-155-214.cn-northwest-1.compute.internal 5.10.186-179.751.amzn2.x86_64 #1 SMP Tue Aug 1 20:51:38 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux/kind bug