search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.8k stars 4.6k forks source link

[Improvement] Kubernetes - execute complex jobs with sidecar containers etc etc #16685

Open giovannidalloglio opened 2 weeks ago

giovannidalloglio commented 2 weeks ago

Search before asking

Description

Hello.

Do you fully support the kubernetes Job api?

In the documentation (https://dolphinscheduler.apache.org/en-us/docs/3.2.2/guide/task/kubernetes) i just see the option to specify a single container.

But various "enterprise" scenarios require the full set of options (https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/).

As example, in our environment we need a "worker" container, and two differente sidecar containers (one of them is the "tunnel" to the DB, etc...)

Is it possibile to run those kind of jobs?

Are you willing to submit a PR?

Code of Conduct

SbloodyS commented 2 weeks ago

cc @Gallardot

Gallardot commented 2 weeks ago

@giovannidalloglio

Currently, not all capabilities and features of K8S Job are fully supported. At present, only the definition of a single container in spec.template.spec.containers is supported; multiple container definitions are not supported. Perhaps in the future, we can allow users to customize the spec.template of the Job, but this is not supported at the moment.

In my personal understanding, Sidecars belong to the general capabilities of the infrastructure and should not be customized by users. Administrator can use tools like gatekeeper to inject more custom features.

cc @fuchanghai @Mighten @SbloodyS @ruanwenjun

SbloodyS commented 2 weeks ago

@giovannidalloglio

Currently, not all capabilities and features of K8S Job are fully supported. At present, only the definition of a single container in spec.template.spec.containers is supported; multiple container definitions are not supported. Perhaps in the future, we can allow users to customize the spec.template of the Job, but this is not supported at the moment.

In my personal understanding, Sidecars belong to the general capabilities of the infrastructure and should not be customized by users. Administrator can use tools like gatekeeper to inject more custom features.

cc @fuchanghai @Mighten @SbloodyS @ruanwenjun

+1