RabbitMQ AWS CloudWatch exporter

A plugin for exporting RabbitMQ logs and metrics collected by the Management Plugin to AWS CloudWatch.

Installing

Download the .ez files from the latest release and copy them into the RabbitMQ plugins directory.

Enable the plugin:

[sudo] rabbitmq-plugins enable rabbitmq_cloudwatch_exporter

Building from Source

Please see RabbitMQ Plugin Development guide.

To build the plugin:

git clone https://github.com/noxdafox/rabbitmq-cloudwatch-exporter.git
cd rabbitmq-cloudwatch-exporter
make dist

Then copy all the *.ez files inside the plugins folder to the RabbitMQ plugins directory and enable the plugin:

[sudo] rabbitmq-plugins enable rabbitmq_cloudwatch_exporter

Exporting RabbitMQ logs to CloudWatch

Together with the plugin is shipped a Lager backend for CloudWatch which allows to export the service logs to AWS CloudWatch Logs.

The handler configuration is explained in details in the RabbitMQ logging configuration guidelines.

The following example configures the Lager backend to export all the logs from info level and above into the RabbitMQ log group. The node names will be used as log stream names.

[{lager,
  [{handlers, [{lager_cloudwatch_backend, [info, "RabbitMQ"]}]}]}].

More details regarding the backend configuration can be found in its README.

AWS Credentials

To resolve AWS credentials, the standard environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are looked up first. Otherwise, the EC2 instance role or ECS task role are employed.

AWS Region

By default the logs are exported to the us-east-1 AWS region. The region can be changed via the environment variable AWS_DEFAULT_REGION.

Exporting RabbitMQ metrics to CloudWatch

AWS Credentials

To resolve AWS credentials, the standard environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are looked up first. Otherwise, the EC2 instance role or ECS task role are employed.

Alternatively, the User can specify them in the rabbitmq.conf file as follows.

cloudwatch_exporter.aws.access_key_id = "AKIAIOSFODNN7EXAMPLE"
cloudwatch_exporter.aws.secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Or in the rabbitmq.config format.

[{rabbitmq_cloudwatch_exporter,
  [{aws, [{access_key_id, "AKIAIOSFODNN7EXAMPLE"},
          {secret_access_key, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"}]}]}].

AWS Region

By default the metrics are published within the us-east-1 AWS region. The region can be changed in the rabbitmq.conf file as follows.

cloudwatch_exporter.aws.region = "us-west-1"

Or in the rabbitmq.config format.

[{rabbitmq_cloudwatch_exporter,
  [{aws, [{region, "us-west-1"}]}]}].

Metrics collection

Metrics are grouped in different categories described below. Each category must be enabled in the configuration in order for its metrics to be exported.

rabbitmq.conf example.

cloudwatch_exporter.metrics.overview.enable = true
cloudwatch_exporter.metrics.vhost.enable = true
# Omitting or setting a group to false is equivalent
cloudwatch_exporter.metrics.node.enable = false

rabbitmq.config example.

[{rabbitmq_cloudwatch_exporter,
  [{metrics, [{overview, [{enable, true}]},
              {vhost, [{enable, true}]},
              {node, [{enable, false}]}]}]}].

If a metrics group is enabled, all its metrics will be exported. The export_metrics configuration parameter allows to further limit which metrics of a specific group to export.

rabbitmq.conf example.

cloudwatch_exporter.metrics.exchange.enable = true
cloudwatch_exporter.metrics.exchange.export_metrics.1 = "PublishIn"
cloudwatch_exporter.metrics.exchange.export_metrics.2 = "PublishOut"

rabbitmq.config example.

[{rabbitmq_cloudwatch_exporter,
  [{metrics, [{exchange, [{enable, true},
                          {export_metrics, ["PublishIn", "PublishOut"]}]}]}]}].

For exchange, queue, connection and channel metrics groups, it is possible to control the names of the entities to be published via regular expressions.

rabbitmq.conf example.

cloudwatch_exporter.metrics.exchange.enable = true
# Only export exchanges which names begin with foo or end with bar
cloudwatch_exporter.metrics.exchange.export_regex = "foo.*|.*bar"

cloudwatch_exporter.metrics.queue.enable = true
# Exclude server-named queues
cloudwatch_exporter.metrics.queue.export_regex = "^(?!amq.gen-.*).*$"

rabbitmq.config example.

[{rabbitmq_cloudwatch_exporter,
  [{metrics, [{exchange, [{enable, true},
                          {export_regex, "foo.*|.*bar"}]},
              {queue, [{enable, true},
                       {export_regex, "^(?!amq.gen-.*).*$"}]}]}]}].

Metrics are exported every minute. The export period can be expressed in seconds via the cloudwatch_exporter.export_period configuration parameter (rabbitmq_cloudwatch_exporter.export_period in rabbitmq.config format). CloudWatch metrics have a standard resolution of one minute. Therefore, publishing more than once per minute will results in the metrics being aggregated on the receiving end. The cloudwatch_exporter.storage_resolution configuration parameter (rabbitmq_cloudwatch_exporter.storage_resolution in rabbitmq.config format) allows to decrease the aggregation period to 1, 5, 10 or 30 seconds.

Lastly, the AWS CloudWatch namespace can be controlled via the cloudwatch_exporter.namespace configuration parameter (rabbitmq_cloudwatch_exporter.namespace in rabbitmq.config format). The default value is RabbitMQ.

Metrics

Metrics are grouped in different categories. All the metrics grouped within a category are characterised by a set of dimensions which help to identify their belonging.

As AWS CloudWatch charges a monthly cost per each metric, the CloudWatch Exporter Plugin exports a limited subset of metrics which are deemed relevant. If a particular metric is missing, please open an Issue Ticket or submit a Pull Request to add it to the exported ones. For the same rationale, the CloudWatch Exporter Plugin only exports metrics from the categories which have been enabled in the configuration file.

Overview

Dimensions

Metrics

VHost

Dimensions

Metrics

Node

Dimensions

Metrics

Exchange

Dimensions

Metrics

Queue

Dimensions

Metrics

Connection

Dimensions

Metrics

Channel

Dimensions

Metrics