Open NewbiZ opened 1 month ago
Nevermind, the DynamicPartitionsDefinition
replacement doesn't work at all in the end.
First, because schedules on dynamic partitions are not really able to take additions / removal into account.
Second, because there is no notion of current partition on dynamic partitions, so dependencies by downstream assets are broken as well.
I am confused as to how partitions are supposed to be used.
I'm coming to the conclusion that the easiest thing to do to overcome the limitations on the number of time window partitions is to emulate them with a config:
DailyConfiguration
object with a single field date
set to None
by default.date
in the config. If it's None
it uses the current date, otherwise it uses the provided date.date
in the config.That should fix the issue of scheduling with dynamic partitions.
What's the use case?
I have an asset that stores daily data on a number of items. I have over a thousand items, and multiple years of data. The asset is currently partitioned by day and item (a
MultiPartitionsDefinition
ofDailyPartitionsDefinition
+DynamicPartitionsDefinition
), leading to an enormous amount of partitions (e.g. 1000 items over just a year = 365,000 partitions).Clearly, this number of partitions does not work with Dagster, the UI becomes unresponsive as soon as I try to display asset lineage or just list existing partitions.
I don't really need to have all these partitions all the time, as long as the underlying files are not removed. The daily partitioning is useful because it allows backfilling, which is a useful support operation, but on a business-as-usual basis, I could just have say the last 30 days and live with that.
This seems like an trivial use case, but I started using Dagster just a week ago so I may have completely missed an obvious way to solve this.
Here is what I tried:
DailyPartitionsDefinition
is more or less built on the idea of being static, you cannot remove partitions from it.PartitionsDefinition
, inspired byTimeWindowPartitionsDefinition
, but skipping the partitions older than now - 30 days. That failed as there is a check in the Dagster code ensuring the only partitions definitions allowed areTimeWindow
/Static
/Dynamic
/Multi
.start_date
ofDailyPartitionsDefinition
, obviously did not workstart_date
of theDailyPartitionsDefinition
be evaluated at runtime, and restart the Dagster daemon every day. This works but is pathetic.DynamicPartitionsDefinition
, and schedule a job to create a new daily partition every day at midnight, and remove the older partition. This is what I'm currently implementing, but is far from ideal. Since the main use case of time window partitions is to allow backfilling, replacing them by a dynamic partition removes the slider in the UI.Ideas of implementation
I would propose either of the following solutions as possible fix:
pruning
parameter to theTimeWindowDefinitionsPartition
that would remove partitions older than the provided value.use_time_slider
(or even more generallyuse_ui_widget=UiWidgetTimeSlider(format="%Y-%m-%d")
) so that we can keep the UI behavior ofTimeWindowPartitionsDefinition
even when usingDynamicPartitionsDefinition
.