Support configuring remote cluster for pulling images

[matthewmichihara]

I'm trying to build a container on GCB, and then deploy it on my local Docker for Desktop Kubernetes cluster, but I seem to be hitting auth issues. The container builds fine, but the Kubernetes pod gets stuck in an error state. I went through https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud-helper and set gcloud as the credential helper via gcloud auth configure-docker, but the problem still seems to persist.

Using this example: https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/master/java/java-hello-world

$ pwd
/Users/michihara/Code/cloud-code-samples/java/java-hello-world

$ skaffold version
v1.8.0

$ gcloud version
Google Cloud SDK 289.0.0
alpha 2020.02.25
app-engine-java 1.9.79
app-engine-python 1.9.90
beta 2020.02.25
bq 2.0.56
cloud-datastore-emulator 2.1.0
core 2020.04.10
gsutil 4.49
minikube
skaffold

$ gcloud auth list
   Credentialed Accounts
ACTIVE  ACCOUNT
*       michihara@google.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

$ gcloud auth configure-docker
WARNING: Your config file at [/Users/michihara/.docker/config.json] contains these credential helper entries:

{
  "credHelpers": {
    "us.gcr.io": "gcloud",
    "asia.gcr.io": "gcloud",
    "marketplace.gcr.io": "gcloud",
    "gcr.io": "gcloud",
    "eu.gcr.io": "gcloud",
    "staging-k8s.gcr.io": "gcloud"
  }
}
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
gcloud credential helpers already registered correctly.

$ skaffold run -p cloudbuild --default-repo gcr.io/chelseamarket/p
Generating tags...
 - java-hello-world -> gcr.io/chelseamarket/p/java-hello-world:latest
Checking cache...
 - java-hello-world: Found Remotely
Tags used in deployment:
 - java-hello-world -> gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997
Starting deploy...
 - deployment.apps/java-hello-world configured
 - service/java-hello-world-external configured
Waiting for deployments to stabilize...
 - deployment/java-hello-world: waiting for rollout to finish: 1 old replicas are pending termination...
 - deployment/java-hello-world failed. Error: could not stabilize within 2m0s: context deadline exceeded.
FATA[0124] 1/1 deployment(s) failed

$ kubectl config current-context
docker-desktop

$ kubectl get all
NAME                                    READY   STATUS             RESTARTS   AGE
pod/java-hello-world-6d47c78bf9-xpm7f   0/1     ImagePullBackOff   0          36s

NAME                                TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
service/java-hello-world-external   LoadBalancer   10.101.0.92   localhost     80:30743/TCP   37s
service/kubernetes                  ClusterIP      10.96.0.1     <none>        443/TCP        45s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/java-hello-world   0/1     1            0           37s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/java-hello-world-6d47c78bf9   1         1         0       37s

$ kubectl describe pod/java-hello-world-6d47c78bf9-xpm7f
Name:           java-hello-world-6d47c78bf9-xpm7f
Namespace:      default
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Fri, 08 May 2020 11:27:08 -0400
Labels:         app=java-hello-world
                app.kubernetes.io/managed-by=skaffold-v1.8.0
                pod-template-hash=6d47c78bf9
                skaffold.dev/builder=google-cloud-build
                skaffold.dev/cleanup=true
                skaffold.dev/deployer=kubectl
                skaffold.dev/profile.0=cloudbuild
                skaffold.dev/run-id=3cffaa33-5716-4371-92ed-43be9f3a1d70
                skaffold.dev/tag-policy=sha256
                skaffold.dev/tail=true
Annotations:    <none>
Status:         Pending
IP:             10.1.1.133
IPs:            <none>
Controlled By:  ReplicaSet/java-hello-world-6d47c78bf9
Containers:
  server:
    Container ID:
    Image:          gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997
    Image ID:
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      PORT:               8080
      JAVA_TOOL_OPTIONS:  -agentlib:jdwp=transport=dt_socket,server=y,address=5005,suspend=n,quiet=y
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-q4ngt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-q4ngt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-q4ngt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From                     Message
  ----     ------     ----               ----                     -------
  Normal   Scheduled  49s                default-scheduler        Successfully assigned default/java-hello-world-6d47c78bf9-xpm7f to docker-desktop
  Normal   BackOff    24s (x3 over 47s)  kubelet, docker-desktop  Back-off pulling image "gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997"
  Warning  Failed     24s (x3 over 47s)  kubelet, docker-desktop  Error: ImagePullBackOff
  Normal   Pulling    13s (x3 over 48s)  kubelet, docker-desktop  Pulling image "gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997"
  Warning  Failed     12s (x3 over 47s)  kubelet, docker-desktop  Failed to pull image "gcr.io/chelseamarket/p/java-hello-world:latest@sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997": rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
  Warning  Failed     12s (x3 over 47s)  kubelet, docker-desktop  Error: ErrImagePull

[tstromberg]

The meat of the error is:

unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Just to test that Docker is in-fact configured properly, can you share the output of:

docker pull gcr.io/chelseamarket/p/java-hello-world

[matthewmichihara]

$ docker pull gcr.io/chelseamarket/p/java-hello-world
Using default tag: latest
latest: Pulling from chelseamarket/p/java-hello-world
24f0c933cbef: Already exists
69e2f037cdb3: Already exists
3e010093287c: Already exists
aebd67d1ef6f: Already exists
d97439fb32b3: Already exists
a0e31a817843: Already exists
5b40285b7db8: Already exists
Digest: sha256:446df95cdad401e4aadf5ccc137648a725ba8d7f25aa0a2b26017f4f262f1997
Status: Downloaded newer image for gcr.io/chelseamarket/p/java-hello-world:latest
gcr.io/chelseamarket/p/java-hello-world:latest

That worked. Had a discussion with @briandealwis about this and the issue seems to be that I don't have ImagePullSecrets configured for this D4D cluster (https://blog.container-solutions.com/using-google-container-registry-with-kubernetes) so this is probably not a Skaffold bug, but I thought I'd keep this issue open for discussion. Feel free to close if you think there's nothing to be done here.

[dgageot]

Yeah, this is not a Skaffold bug. I wonder what Skaffold could do better though. @matthewmichihara any idea?

[matthewmichihara]

Haven't really thought this through but could Skaffold set up this ImagePullSecret for me? Or would you say that is not Skaffold's responsibility

[briandealwis]

I'm not sure if Skaffold is the right place for it, but it does feel like there should be a tool for creating the secret and patching the service account.

[briandealwis]

.I've marked this issue with help wanted. This is not a high-priority issue for the Skaffold team as GKE clusters are already setup to pull images from GCR and Artifact Registry repositories within the same project, and Minikube's gcp-auth addon supports rewriting pods on the fly with GCR+AR pull secrets.

If you are interested in contributing, please create a design proposal for discussion.

---

I should note some complications to creating a solution.

1. Otaining the pull secret: For example, some services request a service account key, others want login/password. Exporting such secrets brings increased risk!
2. Referencing: With the rise of pod creation from custom resources, adding imagePullSecrets to pods can be somewhat difficult.