Multi-image jobs

[johannkm]

Currently dagster's containerized deployments run pipelines using a single image (retrieved from the repository location). In some cases, different solids within a pipeline have vastly different dependencies which are difficult to combine into a single image.

Approach 1: Override the image per solid At runtime, specify image tags for each solid. Dagster will assume that each image contains the same python code for the solid, since the pipeline definition will be ready only from the main image.

Approach 2: Offer better tooling for launching containers from within solids Alternatively, solids could continue to run with a single image but a resource could be provided for launching a command inside another container with a different image. This removes the need to have your pipeline code synced across all your images, and the other images woudn't even need to include dagster or python. The inherent downside is that the compute inside the launched container wouldn't be able to use dagster, e.g. it wouldn't have access to logging (though there are workarounds- the main solid could watch and upload the logs for it).

---

Message from the maintainers:

Excited about this feature? Give it a :thumbsup:. We factor engagement into prioritization.

[nicklofaso]

Very excited for a feature like this!

As I mentioned in a previous comment our pipeline involves steps with vastly different dependencies and programming languages (python, R, Fortran, C++, etc.), so trying to combine all of them into a single image would be extremely difficult.

I'm new to Dagster, but I think Approach 1 provides the most value:

• Approach 1 can leverage IOManagers and Asset Materialization easier than Approach 2 as Dagster code runs on same image as the desired processing (e.g. uploading file to s3). Same for logging and likely other functionality.
• Approach 1 does require a rebuild of each steps image every time the solids/pipeline code is updated. This additional overhead may be worth it to enable the first point.

[darrenhaken]

I am also excited by this as a feature.

I discussed this in Slack but we currently use Airflow extensively with the Pod Operator. We have found developing Docker images around custom operations offered several advantages:

• A custom operation (Solid in Dagster nomenclature) can be developed by a Platform Team and then used by different teams. An example of this - one of my teams developed a Docker image to execute DBT but wrapped it in a custom CLI to apply guard rails to the process. They could do this entirely outside of Airflow.
• Docker images can be executed outside of the Dagster environment
• We can roll out and apply security patches easier with smaller images tailored around a Solid over the entire of Dagster.

I see a question came up around logging. It would be handy if Dagster could parse the stdout/err from the Docker image and then roll that into the Dagster logs. Airflow does something similar to this.

[johannkm]

Current status is that this is possible using https://github.com/dagster-io/dagster/pull/4818, but it's experimental and likely not the end state that we'd like to reach.

[henripal]

Would it be possible to see a minimal working example using https://github.com/dagster-io/dagster/pull/4818 to run a multi-image pipeline?

[evanvolgas]

I'd love to see a working example of a multi-image pipeline too. I've read the changes in https://github.com/dagster-io/dagster/pull/4818 but it's not clear to me how to put them to use.