When I run a job on my EKS cluster using defaults (where max_array_size > 1), which is running v1.21, I see these errors (warnings?):
[redun] Executor[k8s]: Pod redun-job-d64219e107664faab6f1223c52909c0a-array-888pz is missing job-completion-index: {'kubernetes.io/psp': 'rafay-privileged-psp'}
The k8s jobs are all in Error state, and the workflow never finishes because it gets that error.
The k8s executor depends on a feature added in k8s v1.24: https://kubernetes.io/docs/tasks/job/indexed-parallel-processing-static/
When I run a job on my EKS cluster using defaults (where max_array_size > 1), which is running v1.21, I see these errors (warnings?):
[redun] Executor[k8s]: Pod redun-job-d64219e107664faab6f1223c52909c0a-array-888pz is missing job-completion-index: {'kubernetes.io/psp': 'rafay-privileged-psp'}
The k8s jobs are all in Error state, and the workflow never finishes because it gets that error.
We already have code that should be detecting versions less than v1.21 https://github.com/insitro/redun/blob/main/redun/executors/k8s.py#L418 but I think these code path still execute: https://github.com/insitro/redun/blob/main/redun/executors/k8s.py#L478 and https://github.com/insitro/redun/blob/main/redun/executors/k8s.py#L771
To repro, I think you could use minikube to install v1.23 or earlier and then run redun in it. To fix, I think you could remove the warning at https://github.com/insitro/redun/blob/main/redun/executors/k8s.py#L771 and properly handle tasks that are missing that field.