minikube running on google cloud shell is unable to pull image from artifact registry - denied: Unauthenticated request

anant-ca commented 1 month ago

What Happened?

I am running minikube on google cloud shell. When I deploy my app using a command

kubectl apply -k k8/config/app/app-name

the deployment is unable to pull image from artefact registry. This is the error I receive:

Failed to pull image "europe-docker.pkg.dev/sample-dev-xxxxxx/github-cr/org-name/app:v0.0.46": Error response from daemon: Head "https://europe-docker.pkg.dev/v2/sample-dev-xxxxxx/github-cr/org-name/app/manifests/v0.0.46": denied: Unauthenticated request. Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/sample-dev-xxxxxx/locations/europe/repositories/github-cr" (or it may not exist)

Please note the following:

The registry is a remote registry that pulls artefacts from GitHub Container registry. This remote registry is duly authenticated while configuration.
As mentioned above that the above registry is duly authenticated during configuration,
- I am able to pull the image using docker (on google cloud console),
- I am able to pull the image and it deploy on the Cloud Run,
- I am able to pull the image, using the the same kubernetes configuration, and deploy on GKE (google kubernetes engine).
I also tried using the Automated Google Cloud Platform Authentication addon for minikube but it said It seems that you are running in GCE, which means authentication should work without the GCP Auth addon. If you would still like to authenticate using a credentials file, use the --force flag.

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Docker

medyagh commented 1 month ago

anant-ca do you mind trying "gcloud auth login" in cloud shell and see if that fixes the problem (and then enable the gcp-auth addon )

minikube addons enable gcp-auth

anant-ca commented 1 month ago

@medyagh thanks for your prompt response. I tried your suggestion but it didn't solve the problem (minikube logs contain the same error as mentioned above).

Here's the console output for your reference:

anant@cloudshell:~/gh/deploy (sample-dev-434415)$ gcloud auth login

You are already authenticated with gcloud when running
inside the Cloud Shell and so do not need to run this
command. Do you wish to proceed anyway?

Do you want to continue (Y/n)?  y

Go to the following link in your browser, and complete the sign-in prompts:
...

anant@cloudshell:~/gh/deploy (sample-dev-434415)$ gcloud auth list
Credentialed Accounts

ACTIVE: *
ACCOUNT: anant@c-arch.io

To set the active account, run:
    $ gcloud config set account `ACCOUNT`
anant@cloudshell:~/gh/deploy (sample-dev-434415)$ minikube start
😄  minikube v1.33.1 on Ubuntu 22.04 (amd64)
    ▪ MINIKUBE_FORCE_SYSTEMD=true
    ▪ MINIKUBE_HOME=/google/minikube
    ▪ MINIKUBE_WANTUPDATENOTIFICATION=false
✨  Automatically selected the docker driver. Other choices: none, ssh
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.44 ...
💾  Downloading Kubernetes v1.30.0 preload ...
    > gcr.io/k8s-minikube/kicbase...:  481.58 MiB / 481.58 MiB  100.00% 79.67 M
    > preloaded-images-k8s-v18-v1...:  342.90 MiB / 342.90 MiB  100.00% 40.98 M
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ kubelet.cgroups-per-qos=false
    ▪ kubelet.enforce-node-allocatable=""
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
anant@cloudshell:~/gh/deploy (sample-dev-434415)$ minikube addons enable gcp-auth
💡  gcp-auth is an addon maintained by Google. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
❗  It seems that you are running in GCE, which means authentication should work without the GCP Auth addon. If you would still like to authenticate using a credentials file, use the --force flag.
anant@cloudshell:~/gh/deploy (sample-dev-434415)$ kubectl apply -k k8/config/app/user/
namespace/user created
configmap/user-config created
service/user-service created
deployment.apps/user-deployment created
horizontalpodautoscaler.autoscaling/user-hpa created
anant@cloudshell:~/gh/deploy (sample-dev-434415)$ kubectl get all -n user
NAME                                  READY   STATUS         RESTARTS   AGE
pod/user-deployment-999854dbc-jtsr4   0/1     ErrImagePull   0          28s
pod/user-deployment-999854dbc-rmn4w   0/1     ErrImagePull   0          28s

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/user-service   LoadBalancer   10.109.100.15   <pending>     8080:31890/TCP   28s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/user-deployment   0/2     2            0           28s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/user-deployment-999854dbc   2         2         0       28s

NAME                                           REFERENCE         TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/user-hpa   Deployment/user   cpu: <unknown>/80%   2         20        0          28s

medyagh commented 1 month ago

@anant-ca I see , are you sure you are pulling image from Google AR ? or is that "GitHub Container Registry" I think that would require a different auth creds and would need to login to github

anant-ca commented 1 month ago

@medyagh You are right the AR europe-docker.pkg.dev/sample-dev-434415/github-cr is actually a remote registry that pulls artefacts from Github Container Registry. Please note the following though (as mentioned before in my issue details):

The AR is duly authenticated, using the keys from GitHub, while it was configured.
I works well as I am
- able to fetch the artefacts using docker (using google cloud console)
- able to pull the image and deploy it to Cloud Run using the same url
- able to deploy to GKE using the exact same k8 configuration

spowelljr commented 1 month ago

Hi @anant-ca, I think the issue you're experiencing make sense. You have your Docker instance on your host machine authenticated with GitHub Container Registry, but when you start minikube it has its own Docker instance that created and running inside of it and that instance is trying to pull the images, and it isn't authenticated with GitHub Container Registry. We have an addon registry-creds that will pass in registry credentials for GCP, AWS, DockerHub, and a few others, but we don't have this integration for GitHub Container Registry.

How did you authenticate with GitHub Container Registry? If you followed: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-with-a-personal-access-token-classic

You should be able to run minikube ssh and re-run the auth steps in there and then I believe pulling from GitHub Container Registry would work.

anant-ca commented 1 month ago

Hi @spowelljr and @medyagh , Thanks a lot for taking time to explain the reason. Your explanation helped me a lot and I finally managed to resole the issue. In fact, it was not registry-creds but the gcp-auth addon that did the trick. Here'e what I did

Use the command gcloud auth application-default login to generate the ADC.
Copy across the ADC content at the default location ~/.config/gcloud.
Enable the addon by running the command minikube addons enable gcp-auth --force.

The key was to use the --force flag to enable the addon in cloud shell. Initially, when I tried to enable the addon gcp-auth, it gave me the following warning and didn't enable the addon:

anant@cloudshell:~/gh/deploy (sample-dev-434415)$ minikube addons enable gcp-auth
💡  gcp-auth is an addon maintained by Google. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
❗  It seems that you are running in GCE, which means authentication should work without the GCP Auth addon. If you would still like to authenticate using a credentials file, use the --force flag.

You can see that the warning message is misleading. It clearly said that the addon is NOT required in GCE. Wondering if this warning can be removed so that the beginners like me don't get confused.

kubernetes / minikube