Open jparal opened 4 months ago
You mean increase the number of dimensions
Yep, I mean number of partition dimensions. Let me fix the ticket title.
I second this. We are in the process of evaluating dagster for the core orchestration backend of out internal systems. Two dimensions on the partition seems arbitrary and a fundamental blocker which would block us from adopting dagster.
in our use case we have dims, org_id, product_id, scenario_id with depth, 20 - 4 - 5 so around 400 unique partitions
What's the use case?
Our assets would benefit from higher number of partitions. We use dagster for scientific data ETL which are by nature multi-dimensional. I am not sure if an unbounded number of partitions is feasible but ~5 would be sufficient.
Ideas of implementation
n/a
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.