Add Concurrency Control to Matrix Builds

dennisoehme commented 4 months ago

Introduce an option to limit the number of concurrent builds when using Docker Buildx (Bake) to manage matrix builds. This feature would help users control system load and optimize resource utilization during complex build processes.

Description

Currently, Docker Buildx Bake allows users to define and run matrix builds, which can potentially execute a large number of targets in parallel. However, there is no built-in option to limit the concurrency of these builds. This can lead to excessive system load, especially on resource-constrained environments.

Proposed Solution

1) Add a new option to the docker buildx bake command that allows users to specify the maximum number of concurrent build jobs. This option, tentatively named --max-parallel, would control the number of builds executed simultaneously. 2) Add a concurrency field to the target definitions in the docker-bake.hcl file, enabling users to specify the maximum number of concurrent builds for each target group or individual target.

target "build" {
  inherits = ["_common"]
  name = "${PROJECT}-build"
  matrix = {
    "PROJECT" = [...]
  }
  concurrency = 2
}

Use Case(s)

External Download Limits: When Docker builds need to receive downloads from a third-party source that limits the number of parallel requests, controlling concurrency becomes essential. Without such control, exceeding the allowed number of parallel requests can result in build failures or throttling from the third-party service. By specifying concurrency limits in the build configuration, users can ensure compliance with these external constraints while optimizing build efficiency.
Differing Target Loads: Different build targets can produce varying loads on the system. For instance, a target that involves heavy compilation or resource-intensive operations might need to be limited to fewer concurrent builds compared to lighter targets. Specifying concurrency limits per target ensures that resource-intensive builds do not overload the system while allowing lighter builds to proceed in parallel, maximizing overall efficiency.

What do you think, guys? :-)

PS: I know we already have the max-parallelism option in buildkitd (https://github.com/moby/buildkit/issues/1131), but this feature request describes a different purpose.

crazy-max commented 2 months ago

Looks good but don't think this should be specific to matrix but overall invoked build targets so should be only set as flag imo.

thompson-shaun commented 2 months ago

docker / buildx