Use a pool of locks to limit the number of concurrent build on each platform

[jjhelmus]

Use a pool of locks backed by a Git repository to limit the number of concurrent build tasks run on each build platform. Once all locks are acquired additional jobs will wait until a lock in the appropriate pool becomes available and will then start. By default a delay of 10 seconds is used before retrying to acquire a lock although this can be controlled by the retry_delay parameter of the lock resource.

Locks are acquired immediately before the build step and released regardless of the success of the build step.

Locks are only used for the one-off sub-command when the --use_lock_pool argument is included.

This implementation requires the following additions to the concourse configuration files:

• each build worker configuration needs to include a pool_name entry which specifies the name of the pool from which locks will be acquired and released.
• The top level config.yml file must include lock-pool-repo, lock-pool-branch and lock-pool-private-key entries which specify the git repository which controls the resource. Details on the layout of this repository can be found in the pool-resource documentation.

Note that this does not route jobs to a particular workers. This only limits the number of concurrent builds jobs on each platform.

[soapy1]

This looks really awesome! How many concurrent builds can happen on a given worker?

[msarahan]

This is critical foundation, but I don't think this is coupled with a per-worker job limit right now.

Note that this does not route jobs to a particular workers. This only limits the number of concurrent builds jobs on each platform.

We need to tie a given lock to a worker somehow. I'm not sure how easy that might be, since it's tied into concourse's dispatch mechanisms.

[jjhelmus]

Unfortunately the concurrency limit is not per worker but rather per "type" of worker. So you can limit the number of concurrent jobs on all windows workers to say 6 jobs but it is possible that all of these jobs will be scheduled on a single worker rather than spread across all the available workers. This is not ideal but much better than the current situation which allows unlimited jobs to be started on arbitrary workers.

The concurrently per type of worker is set by the number of lock files available in the associated pool in the backing Git repository. For example in the repository I have been testing with the concurrency for the win and linux pools is 3 (there are 3 files in the sub-directories of the linux folder) and 2 for the linux_ppc64le and osx pools. Currently all the lock files in this test repository are unclaimed but these files will move to the claimed sub-directory when jobs acquire a lock and back to the unclaimed sub-directory when they are released. Lock files can be added or removed from the associated pool manually using standard git commands or an automated method could be implemented that added and removed lock files as workers become available or leave.

I would really like to have a method where the lock files could be used to direct a particular build to an available worker but I can not figure how to do this without selecting a particular worker dynamically at run time. It would be possible to select a particular worker when the concourse plan is generated and use a tags step modifier to direct the build task to that worker. The issue with the setup is that if a worker becomes unavailable or is broken all jobs directed to that worker cannot be completed.

[jjhelmus]

We need to tie a given lock to a worker somehow

Dynamic build plans would allow a worker tag to be loaded from the lock file but this feature does not yet exist in concourse, see concourse/concourse#684.

[mingwandroid]

Super cool!