search

GoogleContainerTools / skaffold

Easy and Repeatable Kubernetes Development
https://skaffold.dev/
Apache License 2.0
14.93k stars 1.62k forks source link

DEBU[0018] marking resource failed due to error code STATUSCHECK_IMAGE_PULL_ERR subtask=-1 task=Deploy #8235

Open bestofman opened 1 year ago

bestofman commented 1 year ago

I am trying to deploy my NodeJS application on a local Kubernetes cluster, using skaffold but I get the following result:

INFO[0001] Render completed in 15.042699ms               subtask=-1 task=DevLoop
Tags used in deployment:
 - learnertester/auth -> learnertester/auth:4d1688c1ebbcde95b85b8505b9734837743f007a9a0232aec4c24c6ee6044f07
 - learnertester/ticketing-client -> learnertester/ticketing-client:7e330225fded6bae522511538d55dfcd0f4dc2477166cef0ba13d60393c34edc
 - learnertester/tickets -> learnertester/tickets:20a0b006e4318b47725c03d25fd7adc47d781d096335749b5dc05bbdb8048c80
 - learnertester/orders -> learnertester/orders:98335ecdbaa89f389da83910376e75e0d3caa85474eb49bc1e4d4d5c729b73d6
 - learnertester/expiration -> learnertester/expiration:8c6b05f89e0abe8e6a33da266355cf79713e6bd22d1abda0da5541f24d5d8d9e
 - learnertester/payments -> learnertester/payments:ba1c398b9c2d3572b57a7edae8cded20e662dbfb52862d0c2f5fb10e6a0f584a
DEBU[0001] Local images can't be referenced by digest.
They are tagged and referenced by a unique, local only, tag instead.
See https://skaffold.dev/docs/pipeline-stages/taggers/#how-tagging-works  subtask=-1 task=Deploy
Starting deploy...
DEBU[0001] getting client config for kubeContext: `kubernetes-admin@kubernetes`  subtask=-1 task=DevLoop
DEBU[0001] Running command: [kubectl --context kubernetes-admin@kubernetes get -f - --ignore-not-found -ojson]  subtask=0 task=Deploy
DEBU[0001] Command output: []                            subtask=0 task=Deploy
DEBU[0001] 24 manifests to deploy. 24 are updated or new  subtask=0 task=Deploy
DEBU[0001] Running command: [kubectl --context kubernetes-admin@kubernetes apply -f -]  subtask=0 task=Deploy
 - deployment.apps/auth-depl created
 - service/auth-srv created
 - deployment.apps/auth-mongo-depl created
 - service/auth-mongo-srv created
 - deployment.apps/client-depl created
 - service/client-srv created
 - deployment.apps/expiration-depl created
 - deployment.apps/expiration-redis-depl created
 - service/expiration-redis-srv created
 - ingress.networking.k8s.io/ingress-service created
 - deployment.apps/nats-depl created
 - service/nats-srv created
 - deployment.apps/orders-depl created
 - service/orders-srv created
 - deployment.apps/orders-mongo-depl created
 - service/orders-mongo-srv created
 - deployment.apps/payments-depl created
 - service/payments-srv created
 - deployment.apps/payments-mongo-depl created
 - service/payments-mongo-srv created
 - deployment.apps/tickets-depl created
 - service/tickets-srv created
 - deployment.apps/tickets-mongo-depl created
 - service/tickets-mongo-srv created
INFO[0016] Deploy completed in 14.814 seconds            subtask=-1 task=Deploy
Waiting for deployments to stabilize...
DEBU[0016] getting client config for kubeContext: `kubernetes-admin@kubernetes`  subtask=-1 task=DevLoop
DEBU[0016] getting client config for kubeContext: `kubernetes-admin@kubernetes`  subtask=-1 task=DevLoop
DEBU[0016] checking status deployment/orders-depl        subtask=-1 task=Deploy
DEBU[0016] checking status deployment/auth-mongo-depl    subtask=-1 task=Deploy
DEBU[0016] checking status deployment/nats-depl          subtask=-1 task=Deploy
DEBU[0016] checking status deployment/expiration-redis-depl  subtask=-1 task=Deploy
DEBU[0016] checking status deployment/tickets-depl       subtask=-1 task=Deploy
DEBU[0016] checking status deployment/expiration-depl    subtask=-1 task=Deploy
DEBU[0016] checking status deployment/tickets-mongo-depl  subtask=-1 task=Deploy
DEBU[0016] checking status deployment/auth-depl          subtask=-1 task=Deploy
DEBU[0016] checking status deployment/orders-mongo-depl  subtask=-1 task=Deploy
DEBU[0016] checking status deployment/payments-depl      subtask=-1 task=Deploy
DEBU[0016] checking status deployment/payments-mongo-depl  subtask=-1 task=Deploy
DEBU[0016] checking status deployment/client-depl        subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment orders-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment expiration-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment tickets-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment expiration-redis-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment payments-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment client-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment nats-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment tickets-mongo-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment auth-mongo-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment payments-mongo-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment auth-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment orders-mongo-depl --namespace default --watch=false]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "orders-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "orders-mongo-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "tickets-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "client-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "nats-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "expiration-redis-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "payments-mongo-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "payments-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "expiration-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "auth-mongo-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "auth-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Command output: [Waiting for deployment "tickets-mongo-depl" rollout to finish: 0 of 1 updated replicas are available...
]  subtask=-1 task=Deploy
DEBU[0018] Pod "orders-mongo-depl-d5d848ddf-bbqgv" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "tickets-depl-849c8d456b-vs88q" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "client-depl-775ccc9965-2k9gm" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "expiration-redis-depl-54b5cdbd58-8bzmj" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "orders-depl-586c4b7894-z59cl" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "auth-mongo-depl-5f6657ff85-hm45t" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "payments-depl-6485786b64-t8t8d" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "payments-mongo-depl-7877bc7dc7-4xhtr" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "nats-depl-7699f6bf9c-bfv9n" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] Pod "expiration-depl-7989dc5ff4-lkpvw" scheduled but not ready: checking container statuses  subtask=-1 task=DevLoop
DEBU[0018] marking resource failed due to error code STATUSCHECK_IMAGE_PULL_ERR  subtask=-1 task=Deploy
 - deployment/expiration-depl: container expiration is waiting to start: learnertester/expiration:8c6b05f89e0abe8e6a33da266355cf79713e6bd22d1abda0da5541f24d5d8d9e can't be pulled
    - pod/expiration-depl-7989dc5ff4-lkpvw: container expiration is waiting to start: learnertester/expiration:8c6b05f89e0abe8e6a33da266355cf79713e6bd22d1abda0da5541f24d5d8d9e can't be pulled
 - deployment/expiration-depl failed. Error: container expiration is waiting to start: learnertester/expiration:8c6b05f89e0abe8e6a33da266355cf79713e6bd22d1abda0da5541f24d5d8d9e can't be pulled.
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] pod statuses could not be fetched this time due to following errors occurred context canceled  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] pod statuses could not be fetched this time due to following errors occurred context canceled  subtask=-1 task=Deploy
DEBU[0018] marking resource status check cancelledSTATUSCHECK_USER_CANCELLED  subtask=-1 task=Deploy
DEBU[0018] setting skaffold deploy status to STATUSCHECK_IMAGE_PULL_ERR.  subtask=-1 task=Deploy
Cleaning up...
DEBU[0018] Running command: [kubectl --context kubernetes-admin@kubernetes delete --ignore-not-found=true --wait=false -f -]  subtask=-1 task=DevLoop
 - deployment.apps "auth-depl" deleted
 - service "auth-srv" deleted
 - deployment.apps "auth-mongo-depl" deleted
 - service "auth-mongo-srv" deleted
 - deployment.apps "client-depl" deleted
 - service "client-srv" deleted
 - deployment.apps "expiration-depl" deleted
 - deployment.apps "expiration-redis-depl" deleted
 - service "expiration-redis-srv" deleted
 - ingress.networking.k8s.io "ingress-service" deleted
 - deployment.apps "nats-depl" deleted
 - service "nats-srv" deleted
 - deployment.apps "orders-depl" deleted
 - service "orders-srv" deleted
 - deployment.apps "orders-mongo-depl" deleted
 - service "orders-mongo-srv" deleted
 - deployment.apps "payments-depl" deleted
 - service "payments-srv" deleted
 - deployment.apps "payments-mongo-depl" deleted
 - service "payments-mongo-srv" deleted
 - deployment.apps "tickets-depl" deleted
 - service "tickets-srv" deleted
 - deployment.apps "tickets-mongo-depl" deleted
 - service "tickets-mongo-srv" deleted
INFO[0054] Cleanup completed in 35.7 seconds             subtask=-1 task=DevLoop
DEBU[0054] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0054] Command output: [256
]                        subtask=-1 task=DevLoop
1/12 deployment(s) failed

This is the expiration-depl.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expiration-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: expiration
  template:
    metadata:
      labels:
        app: expiration
    spec:
      containers:
        - name: expiration
          image: learnertester/expiration
          env:
            - name: NATS_CLIENT_ID
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NATS_URL
              value: 'http://nats-srv:4222'
            - name: NATS_CLUSTER_ID
              value: ticketing
            - name: REDIS_HOST
              value: expiration-redis-srv

And this is the expiration-redis-depl.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: expiration-redis-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: expiration-redis
  template:
    metadata:
      labels:
        app: expiration-redis
    spec:
      containers:
        - name: expiration-redis
          image: redis
---
apiVersion: v1
kind: Service
metadata:
  name: expiration-redis-srv
spec:
  selector:
    app: expiration-redis
  ports:
    - name: db
      protocol: TCP
      port: 6379
      targetPort: 6379

Information

apiVersion: skaffold/v2alpha3
kind: Config
deploy:
  kubectl: 
    manifests:
      - ./infra/k8s/*
build:
  local:
    push: false
  artifacts:
    - image: learnertester/auth
      context: auth
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: learnertester/ticketing-client
      context: client
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '**/*.js'
            dest: .
    - image: learnertester/tickets
      context: tickets
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: learnertester/orders
      context: orders
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: learnertester/expiration
      context: expiration
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
    - image: learnertester/payments
      context: payments
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.ts'
            dest: .
bestofman commented 1 year ago

Nobody knows about the problem?

gsquared94 commented 1 year ago

Apologies for no response on this.

From the log message:

DEBU[0017] Running command: [kubectl --context kubernetes-admin@kubernetes rollout status deployment orders-depl --namespace default --watch=false] subtask=-1 task=Deploy

It seems that the value of the kubernetes context is not getting set correctly. Can you verify that the kubernetes context is set correctly(see https://skaffold.dev/docs/environment/kube-context/).

skylight74 commented 1 year ago

I'm also having the same issue with kubeadm, running with the correct context.

filipRisteski commented 1 year ago

I ran into the same issue Ubuntu 22, correct context is set both via yaml and cli, tried both ways but it's showing the same issue.

filipRisteski commented 1 year ago

Here are my thoughts. Upon further investigation it might have something to do with docker contexts. If you want kubernetes on Linux you have to install (additionally) docker desktop, which brings with it another context (daemon). The docker documentation explains this, but I'm not sure if this creates the issue with skaffold. In my organization setting the proper context to docker-desktop and having push: false works on macs. The exact same skaffold.yaml fails on Linux and throws this error. I believe it's because of the existence of double daemons on Linux.

To see if I can get around the error I tried manually changing the docker context, but that didn't get me anywhere.

btsb78 commented 1 year ago

I also faced the same problem. My issue was that I was manually specifying ImagePullPolicy in K8s Deployment config. After removing it, the error was gone.

kurczynski commented 1 year ago

Deleting the cache file under ~/.skaffold/cache fixed the problem for me on v2.5.0.

akpro-io commented 1 year ago

Unfortunately doing that causes all images to be rebuilt even if they were unchanged (and it didn't work)

insanity54 commented 7 months ago

Just putting this here in the hopes that it helps a future visitor. I had this issue and I was rather baffled until I read the previous comments about context. Many thanks for the clues!

Turns out I had made a naming mistake in my .envrc file. (I'm using direnv to set my environment variables.)

$ cat .envrc
kubectl config set-context etmc --cluster=etmc
kubectl config use-context etmc
export KUBECONFIG="$(k3d kubeconfig write etmc)"
export DOCKER_HOST=unix:///var/run/docker.sock

I was using kubectl to set context to etmc. This was wrong because k3d creates it's context names prefixed with k3d-.

$ kubectl config get-contexts
CURRENT   NAME       CLUSTER    AUTHINFO         NAMESPACE
          etmc                                   
*         k3d-etmc   k3d-etmc   admin@k3d-etmc

I looked in the file generated by k3d kubeconfig write etmc and it looks like context is set in there. Both kubectl lines in my .envrc were redundant, then. I changed it to the following and now I'm able to use Skaffold without the deployment errors.

$ cat .envrc
export KUBECONFIG="$(k3d kubeconfig write etmc)"
export DOCKER_HOST=unix:///var/run/docker.sock

EDIT: I think I spoke too soon. This didn't solve my STATUSCHECK_IMAGE_PULL_ERR issue.