search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.62k stars 1.63k forks source link

[feature] Allow setting resource requests and limits for ephemeral storage #11137

Open vanHavel opened 2 months ago

vanHavel commented 2 months ago

Feature Area

/area backend /area sdk

What feature would you like to see?

Allow setting ephemeral requests and limits for resources. This was supported in kfp v1, but is no longer available in kfp v2.

What is the use case or pain point?

Pods might get evicted, if they exceed their ephemeral storage request (which is currently always zero). The k8s scheduler can schedule pods better, based on their ephemeral storage needs.

Is there a workaround currently?

I don't know a workaround. Please let me know, if there is one.

Love this idea? Give it a 👍.

diankasileymane commented 2 months ago

Dear @vanHavel, did you manage to find a workaround ? I am facing the same issue

vanHavel commented 2 months ago

Unfortunately not, I think the best one can do right now is to make sure every pod runs on its own node with enough disk space.

sureshmol commented 1 month ago

Hi @vanHavel I am facing the same issue, Did you find any solution?