search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.66k stars 1.47k forks source link

Enhancements for LaunchPad with Different Materialization Strategies #15909

Open jadenhpark opened 1 year ago

jadenhpark commented 1 year ago

What's the use case?

We recently encountered challenges during our production data asset releases, specifically around handling validation errors in our pipeline. To address these issues, we had to modify the configuration values using the LaunchPad. However, we faced difficulties when trying to override the validations. Due to Dagster’s current limitations in supporting materialization from failure with a modified LaunchPad, we resorted to manually selecting a subset of assets (70 out of 208). This approach was not only cumbersome but also error-prone. We often found ourselves retracing steps after minor missteps, which added complexity to the process.

Ideas of implementation

Enhanced LaunchPad Integration: Allow LaunchPad to support non-default materialization options: "materialize from failure", "materialize from selected", "materialize same steps", and more. This would offer flexibility in managing and recovering data assets, especially in larger pipelines.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

yuhan commented 1 year ago

Thanks for opening up this issue. We're actively exploring ways to consolidate different materialization strategies in the UI. Will keep this issue posted once we have things to share!