kubernetes / kubectl

Issue tracker and mirror of kubectl code
Apache License 2.0
2.89k stars 923 forks source link

how to kubectl wait for job completion regardless of success or failure #1629

Open techlot opened 3 months ago

techlot commented 3 months ago

We are running AKS and use Azure DevOps with kubectl task to deploy our application. Before we deploy the actual application, we are running a data migration job. We want to wait for this job in the Azure DevOps pipeline, however the only option so far is to:

kubectl wait --for=condition=complete --timeout=600s job/somejob

In this case the pipeline task would only exit if the job has completed or the timeout has elapsed. In the case the job fails we are left with just waiting for timeout. I looked at using kubectl wait --for=jsonpath but there is no single value (and i mean single as in singular, not a combination of different values) in the output returned by kubectl that would indicate the job has finished, regardless of whether it succeeded or failed. And due to kubectl not supporting and/or conditions in jsonpath I am not able to evaluate the job completion based on multiple values. The only field that represents true completion is when .status.completionTime is set. I tried just out of curiosity doing kubectl wait --for=jsonpath={.status.completionTime}, and the task did indeed wait for the job once .status.completionTime was set, however docs suggest the condition would be met if the value I look for is True, but obviously completionTime is not. Is there any docs that would indicate how wait --for=jsonpath behaves on the fields that are set dynamically (like .status.completionTime) during the job execution?

k8s-ci-robot commented 3 months ago

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
k8s-ci-robot commented 3 months ago

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
sftim commented 3 months ago

/transfer kubectl

sftim commented 3 months ago

/kind feature /sig cli

techlot commented 3 months ago

Just noticed from the docs that "The completion time is only set when the job finishes successfully.". Hence this will not work in my case when the job fails. Is there other singular value in the job status that represents the completion (finish) of the job, regardless of whether it failed or succeeded?

brianpursley commented 3 months ago

Related issue https://github.com/kubernetes/kubectl/issues/1224

See possible workaround in this comment https://github.com/kubernetes/kubectl/issues/1224#issuecomment-1562346598

Relevant Stack Overflow https://stackoverflow.com/questions/55073453/wait-for-kubernetes-job-to-complete-on-either-failure-success-using-command-line

mpuckett159 commented 3 months ago

/triage accepted There are plans to implement this feature, but until that feature is implemented we recommend that you either implement the scripts in Brian's links, or you can use Brian's plugin as a workaround.

brianpursley commented 3 months ago

As @mpuckett159 mentioned, I created a plugin to help with this.

Install:

kubectl krew update && kubectl krew install wait-job

Usage:

kubectl wait-job somejob

Output with be either: