ploomber / soopervisor

☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.
https://soopervisor.readthedocs.io
Apache License 2.0
45 stars 18 forks source link

Add an option to run a ploomber pipeline in a single pod #30

Closed edublancas closed 9 months ago

edublancas commented 2 years ago

The current integrations run each ploomber task in a separate pod. however, this requires the user to create a shared drive or a File client. Allowing to run everything in a single pod can be useful for simple use cases