search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.17k stars 1.4k forks source link

Allow easily backfilling all failed partitions for different assets with different partitions #22126

Open danielgafni opened 3 months ago

danielgafni commented 3 months ago

What's the use case?

I have 100+ assets with different daily partitions (varying start_date and end_date). Usually, I rely on auto materialization to keep them in sync. However, recently 1 latest partition has failed due to a bug. Right now I can't re-run all the failed Runs or create a backfill for all the assets because it's not possible to select them at once (different partitions). It's too much work to manually go through every asset and materialize the failed partition. Dagster should provide a way of solving this kind of problems.

Edit: I realized I can filter failed Runs by partition and re-run them in bulk of 25. This works (except I have dozens of retries selected too), but I still think there should be other options for doing this

Ideas of implementation

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

ion-elgreco commented 3 months ago

+1 on this, currently it's quite cumbersome to rerun if you have lots of partitions

danielgafni commented 3 months ago

Helpful functionality would be filtering assets based on their status (failed, specific partitions failed, any partitions failed, partitions missing, etc)