search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11k stars 1.38k forks source link

Increase the max number of parition dimensions in MultiPartitionsDefinition #21177

Open jparal opened 4 months ago

jparal commented 4 months ago

What's the use case?

Our assets would benefit from higher number of partitions. We use dagster for scientific data ETL which are by nature multi-dimensional. I am not sure if an unbounded number of partitions is feasible but ~5 would be sufficient.

Ideas of implementation

n/a

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

ion-elgreco commented 4 months ago

You mean increase the number of dimensions

jparal commented 4 months ago

Yep, I mean number of partition dimensions. Let me fix the ticket title.

CrispyCrafter commented 3 months ago

I second this. We are in the process of evaluating dagster for the core orchestration backend of out internal systems. Two dimensions on the partition seems arbitrary and a fundamental blocker which would block us from adopting dagster.

in our use case we have dims, org_id, product_id, scenario_id with depth, 20 - 4 - 5 so around 400 unique partitions