Low code platform for building business apps and workflows in minutes. Supports PostgreSQL, MySQL, MariaDB, MSSQL, MongoDB, Rest API, Docker, K8s, and more 🚀
Describe the feature request
Currently automation can process at a max rate limit of 1 automation per app-service instance, i.e. In a clustered environment with 10 app-services this results in 10 concurrent automations at any one time. This limit can be significantly increased by enabling concurrency on bull queue processing.
There are two options for supporting this:
1) Using the bull concurrency argument of the process function. (Preferred)
Before we can use this option we need to confirm the relationship between the concurrency number and the number of processes spawned. It may not be feasible if a new child process is spawned for every concurrency automation due to the overhead on the instance.
If this is successful we can remove the budibase custom threading from automation processing and rely solely on the bull behaviour that we are not spawning additional threads needlessly
2) Update the existing automation processing to enable reading multiple queue messages at once using in-app logic
It it desirable that concurrent processing is used by default in local and not only something we enable in production systems, including debug support.
Before rolling out it would be best to complete: https://github.com/Budibase/budibase/issues/7560
So that we may adjust the concurrency value to find what works best without affecting the http api
Describe the feature request Currently automation can process at a max rate limit of 1 automation per app-service instance, i.e. In a clustered environment with 10 app-services this results in 10 concurrent automations at any one time. This limit can be significantly increased by enabling concurrency on bull queue processing.
There are two options for supporting this: 1) Using the bull
concurrency
argument of theprocess
function. (Preferred)2) Update the existing automation processing to enable reading multiple queue messages at once using in-app logic
It it desirable that concurrent processing is used by default in local and not only something we enable in production systems, including debug support.
Before rolling out it would be best to complete: https://github.com/Budibase/budibase/issues/7560 So that we may adjust the concurrency value to find what works best without affecting the http api