kubernetes: use activeDeadlineSeconds

[flah00]

k8s jobs can make use of backoffLimit and activeDeadlineSeconds.

• backoffLimit: fail a Job after some amount of retries due to a logical error in configuration etc
  • https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#pod-backoff-failure-policy
• activeDeadlineSeconds: applies to the duration of the job, no matter how many Pods are created. Once a Job reaches activeDeadlineSeconds, the Job and all of its Pods are terminated. The result is that the job has a status with reason: DeadlineExceeded.
  • https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#job-termination-and-cleanup

[bradrydzewski]

We are unable to use the backoffLimit because if a job starts and pipeline starts, and then exits unexpectedly, the pipeline cannot be restarted. This means every retry in the backoff would fail. We could probably enable activeDeadlineSeconds, however, if the job is terminated it will still could leave behind all of the pipeline containers which are running in a separate namespace.

[bradrydzewski]

the previous kubernetes runtime (in this repo) was scrapped. we have a new experimental implementation we are testing which no longer uses jobs so this issue is no longer relevant. code can be found here: https://github.com/drone-runners/drone-runner-kube