Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
183 stars 36 forks source link

[FEATURE] Increase task limit for workflows to 200 #124

Open shwen-gine opened 3 months ago

shwen-gine commented 3 months ago

Is your feature request related to a problem? Please describe. Databricks team have increased the task limit to 200 for react workspace. It would be helpful to have this feature supported in brickflow.

Cloud Information

Describe the solution you'd like Update the _max_tasks_inworkflow variable to 200 image

Describe alternatives you've considered

Additional context Currently, brickflow throws the following error

image