kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.62k stars 1.63k forks source link

[sdk] Pipelines generated from kfp 2.10 ignore cpu/memory requests/limits #11390

Open rimolive opened 2 days ago

rimolive commented 2 days ago

Follow-up from #11374

Environment

Steps to reproduce

  1. Create a python virtual environment
python -m venv venv
source venv/bin/activate
  1. Install kfp 2.10
pip install kfp==2.10
  1. Create the following pipeline with the file name req-test.py
from kfp import dsl

@dsl.component()
def pass_component():
    pass

@dsl.pipeline(name='pipeline-requests')
def pipeline_requests():
    task = pass_component()
    task.set_cpu_request('100m').set_memory_request('100Mi').set_cpu_limit('500m').set_memory_limit('500Mi')
  1. Compile the pipeline
kfp dsl compile --py req-test.py --output req-test.yaml
  1. Upload the pipeline and trigger an execution.

According to @vanHavel user:

After updating to kfp SDK 2.10, we saw pods running with no resources on kfp server 2.3.
It seems that in the generated pipeline spec, the keys for CPU / RAM requests and limits have changed as well (probably in https://github.com/kubeflow/pipelines/pull/11097).

Expected result

The pod should include the resources definition for CPU/Memory, including requests and limits.

Materials and Reference

When compiling the pipeline with 2.10 it renders the following:

        resources:
          resourceCpuLimit: 500m
          resourceCpuRequest: 100m
          resourceMemoryLimit: 500Mi
          resourceMemoryRequest: 100Mi

With older version such as 2.9, it renders the following:

        resources:
          cpuLimit: 0.5
          cpuRequest: 0.1
          memoryLimit: 0.524288
          memoryRequest: 0.1048576

Impacted by this bug? Give it a 👍.

rimolive commented 2 days ago

/assign @rimolive