benpankow
commented
8 months ago Hi Artem, Dagster is built to handle a fairly large variety of use-cases.
Dagster can be deployed in various environments depending on the resources that you require, including platforms like Kubernetes where you can specify resource requirements for each job or each step.
You can also take advantage of tools like Dagster Pipes to execute your code in an external execution environment, which can be useful for machine learning cases where you want to do the heavy compute outside of Dagster but still orchestrate your pipelines in the tool.
What's the use case?
Suppose there's a DAG of several neural networks that execute the task in 1 hour without scaling in total. Is Dagster suitable for serving such complex pipelines in production?
Ideas of implementation
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.