[Question] can we set priorityClassName or priority to pods in kubeflow pipeline?

[garlic-hash]

It's a pure question rather than a bug. Is there any trick to set priorityClassName or priority to pods in pipeline?

The argo workflow seems to support it now: https://github.com/argoproj/argo/pull/1179/files

Here is some comments that kubeflow pipeline doesn't support it because it is "Not supported by Argo". https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/components/structures/kubernetes/v1.py#L408

Thanks!

[Ark-kun]

Unfortunately, this is currently not possible in the SDK. We support all container attributes (via my_task.container.something = ...), but pod attributes require explicit support in the SDK since that's how Argo supports them (e.g. You could not use priorityClassName in Argo until that PR).

What is the scenario where you'd like to use that feature and how critical it is for your project?

[garlic-hash]

Our project runs in on-premises machine. The default pod priority is very low so the kubeflow pipeline will be blocked due to lack of resources. Does the SDK just generate the standard argo workflow yaml so we can upload any valid workflow yaml to kubeflow pipeline frontend? I wonder if there is an ugly workaround that we can use this function to get a workflow dict: https://github.com/kubeflow/pipelines/blob/c4fb79447c50118a3e89f6b30c7d07186f6f812b/sdk/python/kfp/compiler/compiler.py#L754

And modify the workflow outside the SDK before it is uploaded to kubeflow pipeline frontend.

If there is some workaround, this issue is not so critical. We don't use it for production service.

[garlic-hash]

FYI, we had tried "create_workflow" in the kfp 0.1.30. We can get workflow, add podPriorityClassName to it, send it to kubeflow pipeline frontend and run the pipeline well.

We know it is not encouraged so it would be great if new kfp version support priorityClassName but it is not critical for us now.

Thanks!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[garlic-hash]

Should we keep this issue or it is correct to edit and push Argo workflow yaml file to Kubeflow pipeline?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[solarist]

I'd also like to ask for this feature (on prem installation).

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[kavanshukla]

Hi @Ark-kun , any update on this?

the pod spec is not assigning specific priority to the pod (which then defaults to 0). this is preventing the GKE cluster autoscaler from scaling up on preemptible node pools and keeps all the pods in Pending state which halts all the pipelines. my scenario involves spinning up many pipelines in parallel and components/pods within all those pipelines remain unscheduled in Pending state unless I manually provision nodes in the node pool.

having this feature would be very useful. Thanks!

[jbottum]

@kubeflow/wg-pipeline-leads any status on this ...some gcp customers are impacted.

[zijianjoy]

cc @chensun , should we consider increasing the podPriority value to unblock GKE autoscaler issue? Or is it an Argoworkflow question? (It basically means the default podPriority in argoworkflow doesn't work in GKE autoscaler)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tboquet]

Hello, any update on this? I am running heavy parallel runs and I actually want to make sure my jobs are not preempted. It is very explicit in the kubernetes documentation that it is a use case where we want the scheduler to keep everything in the queue instead of preempt nodes.

[stevezkw1998]

FYI, we had tried "create_workflow" in the kfp 0.1.30. We can get workflow, add podPriorityClassName to it, send it to kubeflow pipeline frontend and run the pipeline well.

We know it is not encouraged so it would be great if new kfp version support priorityClassName but it is not critical for us now.

Thanks!

Hi @garlic-hash Thank you for your infomation, however, I try to find where can add podPriorityClassName but fails, would you like to provide more details content or tips, I would appreciate it if you would like to share more

[lethee]

It'd be very useful to use WorkflowSpec.podPriorityClassName. I hope this feature.