Minikube Unable to pull images from AWS ECR on Kubernetes >v1.26.1

itayvolo commented 1 week ago

What Happened?

I'm using Minikube on an AWS EC2 instance (amazonlinux2023) running Kubernetes v1.29.1 with 'Containerd' as the container runtime. The goal is to pull images from a private Amazon ECR repository using the EC2 IAM instance role permissions, without storing explicit credentials.

Everything was working fine with just the IAM instance role until I've decided to upgrade the k8s version from v1.26.1. However, when Minikube attempts to pull these images in pods, it fails with an authentication error (e.g., "no basic auth credentials").

How can I configure Minikube to use the EC2 IAM instance role for pulling images access, just as it worked before?

How to reproduce:

Create a minikube cluster with k8s v1.26.1

Run a pod with an image from private ECR repository

kubectl run <pod_name> --image '<account_id>.dkr.ecr.<region>.amazonaws.com/nginx:latest'

Observe a running pod - successful image pull
Delete minikube cluster
Create a new minikube cluster with k8s version >v1.26.1

Run a pod with an image from private ECR repository

kubectl run <pod_name> --image '<account_id>.dkr.ecr.<region>.amazonaws.com/nginx:latest'

Observe a failed pod - image pull failure

Failed to pull image "<account_id>.dkr.ecr.<region>.amazonaws.com/nginx:latest": failed to pull and unpack image "<account_id>.dkr.ecr.<region>.amazonaws.com/nginx:latest": failed to resolve reference "<account_id>.dkr.ecr.<region>.amazonaws.com/nginx:latest": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials

Operating System

Other

Driver

None (Baremetal)

Helion55 commented 1 week ago

You need to create a secret on your namespace which will store the credentials of your private container registry to pull the image. Try this command to create the secret.

kubectl create secret docker-registry <YOUR-SECRET-NAME>   --docker-server=<URL of the Amazon ECR repository>  --docker-username=<Amazon ECR repository-USERNAME>   --docker-password=<"Amazon ECR repository ACCESS-TOKEN"> or <Amazon ECR repository Password> -n <YOUR-NAMESPACE>

itayvolo commented 1 week ago

@Helion55 As I mentioned in the description, it was working fine before the k8s upgrade with just the iam instance role and without registry-creds, I'm trying to avoid using registry creds and specifying aws key and secret, how can I make it work as it was before?

amitde69 commented 1 week ago

same here, is there a way to make minikube pull images from ECR without configuring anything except an IAM role that allows ECR access ? It worked before k8s 1.27.1

kubernetes / minikube