search

buildkite / buildkite-agent-metrics

A command-line tool (and Lambda) for collecting Buildkite agent metrics
MIT License
66 stars 54 forks source link

Agents in with multiple queues only reporting one queue #117

Open neshmi opened 3 years ago

neshmi commented 3 years ago

I'm particularly interested in being able to observe our various queues and balance agents accordingly. So, as an example, one metric is the "buildkite_queues_total_agent_count" metric. From here, I would be focusing on the percent busy per queue, etc. However, this is one example where the agents seem to only be reporting for a single queue. Each of those queues have several agents assigned to them. The deploy queue actually only has one, and is correct. The other 58 agents share multiple queues, but only show up in the "maint" queue for some reason. So there are a total of 59 agents, and maint is the one queue 58 of the agents share, but each of the other queues should be showing a number of agents as well. I'm using the prometheus backend for this, and Im running version 5.2.1. 

buildkite_queues_total_agent_count{queue="deploy"} 1
buildkite_queues_total_agent_count{queue="build"} 0
buildkite_queues_total_agent_count{queue="k8s"} 0
buildkite_queues_total_agent_count{queue="k8s-build"} 0
buildkite_queues_total_agent_count{queue="elastic"} 0
buildkite_queues_total_agent_count{queue="master"} 0
buildkite_queues_total_agent_count{queue="master-tests"} 0
buildkite_queues_total_agent_count{queue="prod-deploy"} 0
buildkite_queues_total_agent_count{queue="maint"} 58
buildkite_queues_total_agent_count{queue="pipeline"} 0
neshmi commented 3 years ago

Also as a note, I have to specify the queues with the -queue switch to get any queue that isn't maint or deploy to show up.

calvinbui commented 3 years ago

i'm having the same problem. Some queues are missing completely as well.

linus345 commented 1 year ago

Bumping this since we're also seeing the same issue.