Automatic pruning for `TimeWindowPartitionsDefinition`

[NewbiZ]

What's the use case?

I have an asset that stores daily data on a number of items. I have over a thousand items, and multiple years of data. The asset is currently partitioned by day and item (a MultiPartitionsDefinition of DailyPartitionsDefinition + DynamicPartitionsDefinition), leading to an enormous amount of partitions (e.g. 1000 items over just a year = 365,000 partitions).

Clearly, this number of partitions does not work with Dagster, the UI becomes unresponsive as soon as I try to display asset lineage or just list existing partitions.

I don't really need to have all these partitions all the time, as long as the underlying files are not removed. The daily partitioning is useful because it allows backfilling, which is a useful support operation, but on a business-as-usual basis, I could just have say the last 30 days and live with that.

This seems like an trivial use case, but I started using Dagster just a week ago so I may have completely missed an obvious way to solve this.

Here is what I tried:

• Schedule a job that would dynamically remove the daily partitions older than 30 days. That did not work as DailyPartitionsDefinition is more or less built on the idea of being static, you cannot remove partitions from it.
• Create my own subclass of PartitionsDefinition, inspired by TimeWindowPartitionsDefinition, but skipping the partitions older than now - 30 days. That failed as there is a check in the Dagster code ensuring the only partitions definitions allowed are TimeWindow / Static / Dynamic / Multi.
• Dynamically changing the start_date of DailyPartitionsDefinition, obviously did not work
• Have the start_date of the DailyPartitionsDefinition be evaluated at runtime, and restart the Dagster daemon every day. This works but is pathetic.
• Create a DynamicPartitionsDefinition, and schedule a job to create a new daily partition every day at midnight, and remove the older partition. This is what I'm currently implementing, but is far from ideal. Since the main use case of time window partitions is to allow backfilling, replacing them by a dynamic partition removes the slider in the UI.

Ideas of implementation

I would propose either of the following solutions as possible fix:

• Add a pruning parameter to the TimeWindowDefinitionsPartition that would remove partitions older than the provided value.
• Allow marking any partition definition with use_time_slider (or even more generally use_ui_widget=UiWidgetTimeSlider(format="%Y-%m-%d")) so that we can keep the UI behavior of TimeWindowPartitionsDefinition even when using DynamicPartitionsDefinition.

[NewbiZ]

Nevermind, the DynamicPartitionsDefinition replacement doesn't work at all in the end.

First, because schedules on dynamic partitions are not really able to take additions / removal into account.

Second, because there is no notion of current partition on dynamic partitions, so dependencies by downstream assets are broken as well.

I am confused as to how partitions are supposed to be used.

[NewbiZ]

I'm coming to the conclusion that the easiest thing to do to overcome the limitations on the number of time window partitions is to emulate them with a config:

• A DailyConfiguration object with a single field date set to None by default.
• When the asset materializes, it looks at the date in the config. If it's None it uses the current date, otherwise it uses the provided date.
• Backfilling is done by manually materializing the asset with an explicit date in the config.

That should fix the issue of scheduling with dynamic partitions.