Allow wiping partitions

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.

https://dagster.io

Apache License 2.0

11.14k stars 1.4k forks source link

Allow wiping partitions #14749

Open danielgafni opened 1 year ago

danielgafni commented 1 year ago

What's the use case?

Wiping asset partitions can be as useful as wiping normal asset. Would be nice to have it available!

Ideas of implementation

No response

Additional information

[added by @clairelin135] UI performance on the asset graph/asset details page is notably slower when many unused partitions exist in the database. Being able to wipe certain partitions would allow for better UI performance.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

alvarogomezuria commented 10 months ago

Is there any workaround to do so at the moment? I see that in the v1.5.3, it was introduced an option to Report materialization events within the asset view. However, the ideal would be to wipe it instead of mark it as succeed.

Someone knows if it is possible to do it using graphQL? I saw a method called wipeAssets but it seems to wipe it completely.

sryza commented 10 months ago

@alvarogomezuria alas there isn't a workaround at the moment.

samuel-lyu commented 5 months ago

@sryza Hi, why can’t be displayed in paging in the UI? #21002 I also encountered the same problem. The time window of the partition is 5 minutes. I need 520k partitions in the past 5 years, but sometimes I need to recalculate the historical partitions. Modifying the start time of the time partition cannot be achieved.

andoni-garcia-fgp commented 3 weeks ago

Any update here? More granular asset partition management would be really helpful. We also face a similar need as above, where we want to wipe the materialization state of a subset of our history. One additional use case is wiping materialization events of a backfill range for all assets within a DAG. Currently when running a backfill over a DAG, if one asset fails for a given partition, that won't be reflected in the downstream assets. So when you are looking at the downstream asset it appears that all partitions are healthy, when in fact some are in an inconsistent state because the upstream failed. Wiping the materialization events for the backfill range could give better insight into the data integrity for complex DAG backfills.

danielgafni commented 3 weeks ago

Partitions wiping has been implemented in 1.8.0 release

sryza commented 3 weeks ago

Re-opening this because we still need to get it working in open source (it's not a feature that we want to keep Dagster+ only, but it currently it relies on the Dagster+ storage backend to work).

danielgafni commented 3 weeks ago

Oh I didn't know. Sorry about that.