Budibase / budibase

Low code platform for building business apps and workflows in minutes. Supports PostgreSQL, MySQL, MariaDB, MSSQL, MongoDB, Rest API, Docker, K8s, and more 🚀
https://budibase.com
Other
22.71k stars 1.57k forks source link

Concurrent automation processing #7561

Open Rory-Powell opened 2 years ago

Rory-Powell commented 2 years ago

Describe the feature request Currently automation can process at a max rate limit of 1 automation per app-service instance, i.e. In a clustered environment with 10 app-services this results in 10 concurrent automations at any one time. This limit can be significantly increased by enabling concurrency on bull queue processing.

There are two options for supporting this: 1) Using the bull concurrency argument of the process function. (Preferred)

2) Update the existing automation processing to enable reading multiple queue messages at once using in-app logic

It it desirable that concurrent processing is used by default in local and not only something we enable in production systems, including debug support.

Before rolling out it would be best to complete: https://github.com/Budibase/budibase/issues/7560 So that we may adjust the concurrency value to find what works best without affecting the http api

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity.