Limit concurrency per asset in dagster

opensource-observer / oso

Measuring the impact of open source software

https://opensource.observer

Apache License 2.0

53 stars 13 forks source link

Limit concurrency per asset in dagster #1660

Open ravenac95 opened 2 weeks ago

ravenac95 commented 2 weeks ago

What is it?

Currently there are no concurrency limits configured in dagster. So every asset can be running multiple materializations at once. This shouldn't usually be a problem but might cause some unexpected behavior. We should setup our cluster to enable limiting concurrency and thereby preventing issues that may arise because of it.

ravenac95 commented 4 days ago

Docs: https://docs.dagster.io/guides/limiting-concurrency-in-data-pipelines

ryscheng commented 4 days ago

Going to P2 this for now. We should just use #infrastructure in Discord to coordinate manual "locks"