Microk8s - Error: container <container-name> is waiting to start: <image with missing tag> can't be pulled

aliwatters commented 3 years ago

   $ skaffold dev
...
    - pod/server-deployment-5f496cf9c7-2mn6x: container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled

Expected behavior

Image should be pulled/available and everything should work

Actual behavior

deployment fails -- is this user error?

https://github.com/aliwatters/dkc-mutli-k8s/tree/feature-skaffold-minimal (minimal branch)

Information

I'm hitting this error on a unbuntu 20.04 installation, with microk8s and skaffold.

container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled

I'm logged into docker.

Full error is

exiting dev mode because first deploy failed: 1/1 deployment(s) failed
ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold dev
Listing files to watch...
 - aliwatters/dkc-multi-server
Generating tags...
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:90d1283
Checking cache...
 - aliwatters/dkc-multi-server: Found Locally
Tags used in deployment:
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909
Starting deploy...
 - service/server-cluster-ip-service created
 - deployment.apps/server-deployment created
Waiting for deployments to stabilize...
 - deployment/server-deployment: creating container server
    - pod/server-deployment-5f496cf9c7-8dfp8: creating container server
    - pod/server-deployment-5f496cf9c7-hzc2l: creating container server
    - pod/server-deployment-5f496cf9c7-2mn6x: container server is waiting to start: aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 can't be pulled
 - deployment/server-deployment failed. Error: creating container server.
Cleaning up...
 - service "server-cluster-ip-service" deleted
 - deployment.apps "server-deployment" deleted
exiting dev mode because first deploy failed: 1/1 deployment(s) failed

All the relevant files;

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat skaffold.yaml 
apiVersion: skaffold/v2beta11
kind: Config
build:
  artifacts:
    - image: aliwatters/dkc-multi-server
      context: server
      docker:
        dockerfile: Dockerfile.dev
      sync:
        manual:
        - src: '**/*.js'
          dest: .
  local:
    push: false
deploy:
  kubectl:
    manifests:
      - k8s/server-cluster-ip-service.yaml
      - k8s/server-deployment.yaml

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat k8s/server-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: aliwatters/dkc-multi-server
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5000
          env:
            - name: REDIS_HOST
              value: redis-cluster-ip-service
            - name: REDIS_PORT
              value: '6379'
            - name: PGUSER
              value: postgres
            - name: PGHOST
              value: postgres-cluster-ip-service
            - name: PGPORT
              value: '5432'
            - name: PGDATABASE
              value: postgres
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: pgpasswd
                  key: PGPASSWORD

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ cat k8s/server-cluster-ip-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 5000
      targetPort: 5000

And the microk8s config

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ more ~/.kube/config 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBVENDQWVtZ0F3SUJBZ0lKQUx
aR3B0NDZ... snip ..EhqYThFZjBsMzVFdmxFa1pJeGpwOHh5NEVLUHMKTFpCbDB3TT0KLS0tLS1FTkQgQ0VSV
ElGSUNBVEUtLS0tLQo=
    server: https://127.0.0.1:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: VDlqa3BnWlR4Q3UzaHhpQnFvamlkbDdBRk9aMzdBcDkvMFhZT21ad1VqND0K

Versions

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ snap list
Name                  Version                     Rev    Tracking         Publisher    Notes
kubectl               1.20.2                      1811   latest/stable    canonical✓   classic
microk8s              v1.20.1                     1910   1.20/stable      canonical✓   classic

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.1-34+e7db93d188d0d1", GitCommit:"e7db93d188d0d12f2fe5336d1b85cdb94cb909d3", GitTreeState:"clean", BuildDate:"2021-01-11T23:48:42Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.1-34+e7db93d188d0d1", GitCommit:"e7db93d188d0d12f2fe5336d1b85cdb94cb909d3", GitTreeState:"clean", BuildDate:"2021-01-11T23:50:46Z", GoVersion:"go1.15.6", Compiler:"gc", Platform:"linux/amd64"}

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold version
v1.18.0

Steps to reproduce the behavior

https://github.com/aliwatters/dkc-mutli-k8s/tree/feature-skaffold-minimal
skaffold dev

Additonal info

I posted on stackoverflow -- and received a comment suggest I run through the private image steps https://stackoverflow.com/questions/65964535/skaffold-and-microk8s-container-server-is-waiting-to-start-image-cant-be?noredirect=1#comment116673535_65964535

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-pod-that-uses-your-secret -- seems like a red herring as the image is not in a private registry.

$ kubectl get secret regcred --output=yaml
apiVersion: v1
data:
  .dockerconfigjson: ewoJ<snip>KfQ==
kind: Secret
metadata:
  creationTimestamp: "2021-02-01T15:36:02Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:.dockerconfigjson: {}
      f:type: {}
    manager: kubectl-create
    operation: Update
    time: "2021-02-01T15:36:02Z"
  name: regcred
  namespace: default
  resourceVersion: "2368742"
  selfLink: /api/v1/namespaces/default/secrets/regcred
  uid: 7f292deb-6d6a-4ea8-97c4-b3c2966d7abc
type: kubernetes.io/dockerconfigjson

Added

      imagePullSecrets:
        - name: regcred

to the end of my k8s/server-deployment.yaml -- made no difference.

Thanks for looking!

Added: https://github.com/GoogleContainerTools/skaffold/issues/5334 - related?

MarlonGamez commented 3 years ago

hi @aliwatters, thanks for opening this issue. I was able to reproduce this issue using your provided repo on MacOS as well. I'll look into this and see if I can find what's up :)

MarlonGamez commented 3 years ago

Hey again @aliwatters, I used this microk8s documentation to solve this issue: https://microk8s.io/docs/registry-images

What is happening is that microk8s doesn't by default know about the image built by your local docker daemon, so you'll have to import it into microk8s. I was able to do so using the following commands:

docker save aliwatters/dkc-multi-server > aliwatters-dkc-multi-server.tar
multipass transfer aliwatters-dkc-multi-server.tar microk8s-vm:aliwatters-dkc-multi-server.tar
microk8s ctr image import aliwatters-dkc-multi-server.tar

I also had to run microk8s enable dns, however you might have done that already. Please let me know if this helps!

Essentially the issue is that we don't have special logic to handle loading images into microk8s clusters like we do with kind and k3d currently. I don't think our team will be able to prioritize this right now, but if you'd like to potentially contribute to the project I can give some pointers!

Most of the logic that handle this special casing can be found here

https://github.com/GoogleContainerTools/skaffold/blob/af3a492bf9b5a0591e78dd9aaaa51995dce22d0e/pkg/skaffold/config/util.go#L196-L241

and here

https://github.com/GoogleContainerTools/skaffold/blob/af3a492bf9b5a0591e78dd9aaaa51995dce22d0e/pkg/skaffold/runner/deploy.go#L105-L130

we'd essentially just have to add some more helper functions to detect microk8s clusters and then helper functions to load the images into microk8s :)

aliwatters commented 3 years ago

Thank you Marlon!

Yes -- this worked -- and I now understand what is happening!

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ docker save aliwatters/dkc-multi-server > aliwatters-dkc-multi-server.tar

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ multipass transfer aliwatters-dkc-multi-server.tar microk8s-vm:aliwatters-dkc-multi-server.tar
transfer failed: instance "microk8s-vm" does not exist

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ microk8s ctr image import aliwatters-dkc-multi-server.tar
[sudo] password for ali: 
unpacking docker.io/aliwatters/dkc-multi-server:1dc48c4 (sha256:5a28a5bb0010b1e296d496b0fe3563aea263e8abad2152c9c64bdf16956d1a3f)...done
unpacking docker.io/aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909 (sha256:5a28a5bb0010b1e296d496b0fe3563aea263e8abad2152c9c64bdf16956d1a3f)...done
unpacking docker.io/aliwatters/dkc-multi-server:90d1283 (sha256:5a28a5bb0010b1e296d496b0fe3563aea263e8abad2152c9c64bdf16956d1a3f)...done
unpacking docker.io/aliwatters/dkc-multi-server:e9d99b8 (sha256:5a28a5bb0010b1e296d496b0fe3563aea263e8abad2152c9c64bdf16956d1a3f)...done
unpacking docker.io/aliwatters/dkc-multi-server:f450a07 (sha256:5a28a5bb0010b1e296d496b0fe3563aea263e8abad2152c9c64bdf16956d1a3f)...done

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ skaffold dev
Listing files to watch...
 - aliwatters/dkc-multi-server
Generating tags...
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:f450a07
Checking cache...
 - aliwatters/dkc-multi-server: Found Locally
Tags used in deployment:
 - aliwatters/dkc-multi-server -> aliwatters/dkc-multi-server:7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909
Starting deploy...
 - service/server-cluster-ip-service created
 - deployment.apps/server-deployment created
Waiting for deployments to stabilize...
 - deployment/server-deployment: Failed: Error: secret "pgpasswd" not found
    - pod/server-deployment-85c475cf6b-krzmz: Failed: Error: secret "pgpasswd" not found
... # Note: app errors -- all expected and good
cleanup

I pasted the multipass command blindly and transfer failed: instance "microk8s-vm" does not exist -- so would just have to adjust that. But skaffold dev worked at that point.

Images are in my local docker repository

ali@stinky:~/git/dkc-multi-k8s (feature-skaffold-minimal)$ docker images
REPOSITORY                                    TAG                                                                IMAGE ID            CREATED             SIZE
...
localhost:32000/dkc-multi-server              7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909   7b9b7d87ab81        7 days ago          131MB
aliwatters/dkc-multi-server                   7b9b7d87ab81483cdb9b1227780d68dec815fe271aea2f9fe3a0e65090d90909   7b9b7d87ab81        7 days ago          131MB
...

I would love to contribute to this -- but I'm right at the very edge of my skill set in this area. I'll play around though.

aliwatters commented 3 years ago

Raised a draft PR -- is the general approach correct?

https://github.com/GoogleContainerTools/skaffold/pull/5340

(need to fix a config bug, add tests and confirm the internal command used by microk8s to import)

MarlonGamez commented 3 years ago

Thanks so much! The general approach looks good :) We can move our discussion to your PR for now

aaron-prindle commented 3 years ago

Friendly ping, I see that the fix for this Issue - #5340 is in progress. Replying here to see if if there is anything that we can do to provide some assistance.

tejal29 commented 3 years ago

Unfortunately, we closed #5340 due to inactivity. Any one interested in pursuing this?

bestofman commented 1 year ago

Unfortunately, we closed #5340 due to inactivity. Any one interested in pursuing this?

I got similar issue using Kubeadm https://github.com/GoogleContainerTools/skaffold/issues/8235

GoogleContainerTools / skaffold