GoogleContainerTools / skaffold

Easy and Repeatable Kubernetes Development
https://skaffold.dev/
Apache License 2.0
14.94k stars 1.62k forks source link

Precedence given to progressDeadline even if its value greater than statusCheckDeadlineSeconds #9519

Open tomekblazejowski opened 3 weeks ago

tomekblazejowski commented 3 weeks ago

Expected behavior

According to this in docs: https://skaffold.dev/docs/status-check/#configuring-timeout-for-status-check in case that both statusCheckDeadlineSeconds and Deployment.spec.progressDeadlineSeconds are set in skaffold.yaml, precedence is given to Deployment.spec.progressDeadline only if it is less than statusCheckDeadlineSeconds.

Actual behavior

When the following is set in deploy section of skaffold.yaml: statusCheckDeadlineSeconds: 5 and the k8s deployment contains: progressDeadlineSeconds: 60 (more than the status-check timeout) then deployments that lasts more than 5s are not interrupted by skaffold

DEBU[0011] Running command: [kubectl --context microk8s rollout status deployment nodejs-app-deployment --namespace default --watch=false]  subtask=0 task=Deploy
DEBU[0011] Command output: [deployment "nodejs-app-deployment" successfully rolled out
]  subtask=0 task=Deploy
 - deployment/nodejs-app-deployment is ready.
Deployments stabilized in 11.051 seconds
INFO[0011] Deploy completed in 11.614 seconds            subtask=-1 task=Deploy
DEBU[0011] getting client config for kubeContext: `microk8s`  subtask=-1 task=DevLoop

Having above settings once you comment progressDeadlineSeconds: 60 in k8s deployment status-check interrupts after 5s

DEBU[0006] Running command: [kubectl --context microk8s rollout status deployment nodejs-app-deployment --namespace default --watch=false]  subtask=0 task=Deploy
DEBU[0006] pod statuses could not be fetched this time due to following errors occurred client rate limiter Wait returned an error: context deadline exceeded  subtask=0 task=Deploy
DEBU[0006] marking resource failed due to error code STATUSCHECK_USER_CANCELLED  subtask=0 task=Deploy
DEBU[0006] setting skaffold deploy status to STATUSCHECK_USER_CANCELLED.  subtask=0 task=Deploy
DEBU[0006] Running command: [tput colors]                subtask=-1 task=DevLoop
DEBU[0006] Command output: [256
]                        subtask=-1 task=DevLoop
1/1 deployment(s) failed

Ergo the precedence is given to Deployment.spec.progressDeadline even if it is greater than statusCheckDeadlineSeconds.

Information

[...]
deploy:
  helm:
    releases:
      - name: nodejs-app
        chartPath: helm/nodejs-app
        setValueTemplates:
          image.repository: "{{.IMAGE_REPO_nodejs_app}}"
          image.tag: "{{.IMAGE_TAG_nodejs_app}}@{{.IMAGE_DIGEST_nodejs_app}}"
        setValues:
          env: "local"
        valuesFiles:
          - helm/nodejs-app/values.yaml
        overrides:
          global: { }
            statusCheckDeadlineSeconds: 5
            tolerateFailuresUntilDeadline: true
[...]