load balance jobs by agent attribute

[boz]

We have 3 machines each running 3 buildkite agents. I'd like to spread jobs across all three machines evenly: if a build has three jobs that run in parallel, each machine would run one job. This doesn't seem to be the case now - jobs tend to be handled by agents running on the same machine before agents from another machine are used.

It'd be great if we could use an agent attribute to identify the machine that the agent is running on (say, aws:instance-id), and spread jobs evenly among agents with different values of that attribute.

[frangz]

Any news about this? It's currently a problem for us too :(

[toolmantim]

Sorry for the lack of reply @boz and @frangz! There's a pattern to accomplish that now, and that's to use the priority agent configuration setting: https://buildkite.com/docs/agent/v3/prioritization

If you have 3 agents per machine, for each agent on a machine you'd set the priority 3, 2 and 1. And then Buildkite will load balance across the agents, because it'll assign the high priority agents first.

Let me know if that solves it for you, it's been used by many others and we should include it in the documentation.

[boz]

@toolmantim I tried to go down that route by modifying https://github.com/buildkite/elastic-ci-stack-for-aws but hit a dead end trying to get the counter for agent instance number. I should probably try taking another crack at it. Cheers.

[toolmantim]

Ah, @lox might know the best answer to get it working with the Elastic Stack

[lox]

I think it's implemented out of the box in the latest Elastic Stack! https://github.com/buildkite/elastic-ci-stack-for-aws/releases via https://github.com/buildkite/elastic-ci-stack-for-aws/pull/539