Limit concurrency of builds (Rate limit builds)

go-vela / community

Community Information for Vela (Target's official Pipeline Automation Framework)

https://go-vela.github.io/docs/

Apache License 2.0

22 stars 3 forks source link

Limit concurrency of builds (Rate limit builds) #27

Closed jbrockopp closed 2 years ago

jbrockopp commented 4 years ago

Description

Original story: https://github.com/go-vela/server/issues/123

Enable the ability to limit the concurrent running vela pipelines for a given repository and/or given branches within the repository.

Some extra thoughts needed for considerations:

should the setting be defined within the .vela.yml and/or settings page of the UI?
should builds be queued if the limit is reached or have the webhook return some error?

Value

This is primarily helpful for repositories that are implementing deployments, and having multiple run at the same time can cause unintended behavior.

Definition of Done

Users are able to define the concurrency of running vela pipelines within the API.

Effort (Optional)

1 week

Impacted Personas (Optional)

users

stale[bot] commented 4 years ago

This issue has been automatically marked as stale due to no recent activity for the last 60 days. It will be closed if no further activity occurs in the next 30 days. Thank you for your contributions!

dan-kirberger commented 4 years ago

Would this be strictly a user feature? Or could a separate admin focused functionality be considered?

In a shared environment there is the risk of noisy neighbors hogging job slots with many/slow builds. It could be valuable if workers avoided grabbing builds for orgs or repos that have too many active builds going (unless there is nothing else queued up).

Our team likes to shotgun several builds in parallel to run our tests (with each build running a few test suites in parallel), would be nice to be able to do this without a guilty conscience ;)

kneal commented 4 years ago

@dan-kirberger I think this would be focused on the admin perspective to ensure the users can't abuse the platform by pumping high amounts of builds. There definitely use cases for X org/repo to need more a higher throughput than others to account for variance in repository activity

kneal commented 4 years ago

Overal Goals

limit could be set at the org and/or repo level.
implement a way to be available to workers to determine how many builds are currently running for an org
implement a way for the worker to know what the (org/repo) limit is
implement a way to set the limit
worker has access to the number of builds for an org
worker can find the next available build in the queue that is not at the limit for that org
implement a method to prioritize a build(s) / override rate-limiting for an org if build(s) are required to run