search

buildkite / buildkite-agent-metrics

A command-line tool (and Lambda) for collecting Buildkite agent metrics
MIT License
67 stars 50 forks source link

readme

Buildkite Agent Metrics

A command-line tool for collecting Buildkite agent metrics, focusing on enabling auto-scaling. Currently AWS Cloudwatch, StatsD, Prometheus, Stackdriver and New Relic are supported.

Build status

Installing

Either download the latest binary from Github Releases or install with:

go install github.com/buildkite/buildkite-agent-metrics/v5@latest

Running

Several running modes are supported. All of them require an Agent Registration Token, found on the Buildkite Agents page.

Running as a Daemon

The simplest deployment is to run as a long-running daemon that collects metrics across all queues in an organization.

buildkite-agent-metrics -token abc123 -interval 30s

Restrict it to a single queue with -queue:

buildkite-agent-metrics -token abc123 -interval 30s -queue my-queue

Restrict it to multiple queues by repeating -queue:

buildkite-agent-metrics -token abc123 -interval 30s -queue my-queue1 -queue my-queue2

When using clusters, you can pass a cluster registration token to gather metrics only for that cluster:

buildkite-agent-metrics -token clustertoken ...

You can repeat -token to gather metrics for multiple clusters:

buildkite-agent-metrics -token clusterAtoken -token clusterBtoken ...

Running as an AWS Lambda

An AWS Lambda bundle is created and published as part of the build process. The lambda will require the cloudwatch:PutMetricData IAM permission.

It requires a provided.al2 environment and respects the following env vars:

To adjust timeouts, and connection pooling in the HTTP client use the following env vars:

To assist with debugging the following env vars are provided:

Additionally, one of the following groups of environment variables must be set in order to define how the Lambda function should obtain the required Buildkite Agent API token:

Option 1 - Provide the token(s) as plain-text

Option 2 - Retrieve token from AWS Systems Manager

Note: Parameters stored as String and SecureString are currently supported.

Option 3 - Retrieve token from AWS Secrets Manager

Note 1: Both SecretBinary and SecretString are supported. In the case of SecretBinary, the secret payload will be automatically decoded and returned as a plain-text string.

Note 2: BUILDKITE_AGENT_SECRETS_MANAGER_JSON_KEY can be used on secrets of type SecretBinary only if their binary payload corresponds to a valid JSON object containing the provided key.

aws lambda create-function \
  --function-name buildkite-agent-metrics \
  --memory 128 \
  --role arn:aws:iam::account-id:role/execution_role \
  --runtime provided.al2 \
  --zip-file fileb://handler.zip \
  --handler handler

Running as a Container

You can build a docker image for the buildkite-agent-metrics following:

docker build -t buildkite-agent-metrics .

This will create a local docker image named as buildkite-agent-metrics that you can tag and push to your own registry.

You can use the command-line arguments in a docker execution in the same way as described before:

docker run --rm buildkite-agent-metrics -token abc123 -interval 30s -queue my-queue

Supported command line flags

$ buildkite-agent-metrics --help
Usage of buildkite-agent-metrics:
  -backend string
        Specify the backend to use: cloudwatch, newrelic, prometheus, stackdriver, statsd (default "cloudwatch")
  -cloudwatch-dimensions string
        Cloudwatch dimensions to index metrics under, in the form of Key=Value, Other=Value
  -cloudwatch-region string
        AWS Region to connect to, defaults to $AWS_REGION or us-east-1
  -debug
        Show debug output
  -debug-http
        Show full http traces
  -dry-run
        Whether to only print metrics
  -endpoint string
        A custom Buildkite Agent API endpoint (default "https://agent.buildkite.com/v3")
  -interval duration
          Update metrics every interval, rather than once
  -cloudwatch-high-resolution
        If `-interval` is less than 60 seconds send metrics to CloudWatch as [High-Resolution Metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#high-resolution-metrics) which incurs additional charges.
  -max-idle-conns int
        Maximum number of idle (keep-alive) HTTP connections for Buildkite Agent API. Zero means no limit, -1 disables connection reuse. (default 100)
  -newrelic-app-name string
        New Relic application name for metric events
  -newrelic-license-key string
        New Relic license key for publishing events
  -prometheus-addr string
        Prometheus metrics transport bind address (default ":8080")
  -prometheus-path string
        Prometheus metrics transport path (default "/metrics")
  -queue value
        Specific queues to process
  -quiet
        Only print errors
  -stackdriver-projectid string
        Specify Stackdriver Project ID
  -statsd-host string
        Specify the StatsD server (default "127.0.0.1:8125")
  -statsd-tags
        Whether your StatsD server supports tagging like Datadog
  -timeout int
        Timeout, in seconds, for HTTP requests to Buildkite API (default 15)
  -token value
        Buildkite Agent registration tokens. At least one is required. Multiple cluster tokens can be used to gather metrics for multiple clusters.
  -version
        Show the version

Backends

By default metrics will be submitted to CloudWatch but the backend can be switched to StatsD or Prometheus using the command-line argument -backend statsd or -backend prometheus respectively.

Cloudwatch

The Cloudwatch backend supports the following arguments:

StatsD (Datadog)

The StatsD backend supports the following arguments:

Prometheus

The Prometheus backend supports the following arguments:

Stackdriver

The Stackdriver backend supports the following arguments:

The New Relic backend supports the following arguments:

Upgrading from v2 to v3

  1. The -org argument is no longer needed
  2. The -token argument is now an Agent Registration Token — the same used in the Buildkite Agent configuration file, and found on the Buildkite Agents page.
  3. Build and pipeline metrics have been removed, focusing on agents and jobs by queue for auto–scaling. If you have a compelling reason to gather build or pipeline metrics please continue to use the previous version or open an issue with details.

Development

This tool is built with Go 1.20+ and assumes Go Modules by default.

You can build and run the binary tool locally with Go installed:

go run *.go -token [buildkite agent registration token]

Currently this will publish metrics to Cloudwatch under the custom metric prefix of Buildkite, using AWS credentials from your environment. The machine will require the cloudwatch:PutMetricData IAM permission.

The token package

It is an abstraction layer enabling the retrieval of a Buildkite Agent API token from different sources.

The current supported sources are:

Tests

All the tests for AWS dependant resources require their corresponding auto-generated mocks. Thus, before running them, you need to generate such mocks by executing:

go generate token/secretsmanager_test.go
go generate token/ssm_test.go

Metrics

The following metrics are gathered when no specific queue is supplied:

Buildkite > (Org) > RunningJobsCount
Buildkite > (Org) > ScheduledJobsCount
Buildkite > (Org) > UnfinishedJobsCount
Buildkite > (Org) > WaitingJobsCount
Buildkite > (Org) > IdleAgentsCount
Buildkite > (Org) > BusyAgentsCount
Buildkite > (Org) > BusyAgentPercentage
Buildkite > (Org) > TotalAgentsCount

Buildkite > (Org, Queue) > RunningJobsCount
Buildkite > (Org, Queue) > ScheduledJobsCount
Buildkite > (Org, Queue) > UnfinishedJobsCount
Buildkite > (Org, Queue) > WaitingJobsCount
Buildkite > (Org, Queue) > IdleAgentsCount
Buildkite > (Org, Queue) > BusyAgentsCount
Buildkite > (Org, Queue) > BusyAgentPercentage
Buildkite > (Org, Queue) > TotalAgentsCount

When a queue is specified, only that queue's metrics are published.

We send metrics for Jobs in the following states:

License

See LICENSE.md (MIT)