Open ecerulm opened 3 years ago
Why do you think about creating a factory method?
def limits(cpu, memory):
return {
"pod_override": k8s.V1Pod(
spec=k8s.V1PodSpec(
containers=[
k8s.V1Container(
name="base",
resources=k8s.V1ResourceRequirements(
limits={
"memory": memory,
"cpu": cpu
}
)
)
]
)
)
Usage:
task1 = PythonOperator(
task_id="id1",
python_callable=mycallable,
dag=dag1,
executor_config=limits('1', '6Gib')
)
There is already a resources
argument to BaseOperator -- if we want some simple way to set those limits we could use that. I think that parameter is currently otherwise unused (it was used for the old Mesos and cgroup executors)
Description
Currently, if you want to change the kubernetes resources allocated to an operator you need to something like
which works, but I think that is very verbose and for a (in my opinion) common case. I would prefer a more direct syntax like
Alternatively, maybe a documentation change to
would make this (IMHO common) task easier
Use case / motivation
It think the figuring out the
executor_config
>pod_override
and thekubernetes.client.models
api just to increase the memory limit for a tasks was kind of hard. I guess a lot of people would benefit fromAre you willing to submit a PR?
Related Issues