Choosing only a subset of metrics to publish

[mumumilk]

I'm currently using this library and would be really nice to choose between a subset of metrics to publish on every collection.

Why? Well, my broker has something like 300 queues/exchanges. Even when i choose only a specific collection (e.g "queue"), i'm publishing about 3k+ metrics, which most of them i don't need at the moment, to AWS each time.

So, maybe a solution to this kind of problem could be publishing only a subset of metrics (e.g "MessagesReady").

In the config file could be something like this (with default to all supported): cloudwatch_exporter.metrics.queue = ["messages_ready","messages_unacknowledged"]

Although i never worked with elixir, i'll try to submit a PR soon.

[noxdafox]

I was thinking about something similar while developing this plugin. The main challenge is finding a proper syntax for inclusions/exclusions. The above example you provided for example would not work as RMQ sysctl syntax is not supporting lists.

Your above example would rather look like:

cloudwatch_exporter.metrics.queue.1 = messages_ready
cloudwatch_exporter.metrics.queue.2 = messages_unacknowledged

You can imagine how tedious would become the above syntax when extended to all metrics available.

[iTaybb]

I believe that the default should be to include all of them. If the cloudwatch_exporter.metrics.queue list is defined, only those messages will be used.

[oytuntez]

Oh yes, we just published 400K metrics (stomp websocket) by mistake with a cost of ~$440. I am disabling queue metrics now – filter would help a lot. Some important queues always have the same name, I just need to get rid of dynamic queues appearing disappearing.

[noxdafox]

Sorry to hear that.

Avoiding the above is the reason why I opted for not publishing anything by default. Nevertheless, there's definitely room for improvement.

What about a regex based whitelist for entities such as queues, exchanges, connections and channels?

Something like

cloudwatch_exporter.metrics.1 = queue

cloudwatch_exporter.export_regex.queue = "foo.*|bar.*|.*baz"

If the filter is provided, only those entities which name matches the regex will be published into CW. Omitting the filter would be pretty much the equivalent of:

cloudwatch_exporter.export_regex.queue = ".*"

[oytuntez]

@noxdafox I think that would be very helpful, yes!

[noxdafox]

Release 0.3.0 adds support for regex based entities selection. You can see a description with examples in the README.

[iTaybb]

Thank you mate. much appreciated.

On Sat, Nov 30, 2019 at 11:06 PM Matteo Cafasso notifications@github.com wrote:

Release 0.3.0 adds support for regex based entities selection. You can see a description with examples in the README.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/noxdafox/rabbitmq-cloudwatch-exporter/issues/6?email_source=notifications&email_token=AA3SSBWXL4HHNA6FFKWRLIDQWLITVA5CNFSM4HX2IAOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFQT6BY#issuecomment-560021255, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3SSBVSRL7J4W5UY6I27NTQWLITVANCNFSM4HX2IAOA .

[noxdafox]

Closing the issue, please re-open if the solution is not satisfying.

[AlexRomashevski]

Issue is still valid.

In our case we have a lot of Queues in Bus, which are satisfiex by regex filter. The autoscaling for consumes is used just only one Metric, whcih is Queue.Messages. But with current version we have a lot all Queue Metrics which increase CloudWatch price a lot for business case with a lot of Messages in the Bus.

Still suggest to have something like this to manage cost in CloudWatch properly: cloudwatch_exporter.metrics.queue = ["messages"] - just specify exactly metrics which are used for AutoScaling.

Thanks you, Alex R.

[Pilskalns]

Hi,

I am facing the same issue - having a lot of exchanges/queues, but need only 1-2 metrics from each. Would it be possible to add also a regex filter for metrics names?

IMO it would be fine also as proposed before with the list:

cloudwatch_exporter.metrics.queue.1 = messages_ready cloudwatch_exporter.metrics.queue.2 = messages_unacknowledged

It's something that you would set up once and not touch that often, I think it's totally acceptable with extra lines in the sake of reduced messages posted to CW.

P.S. Thanks for the great plugin, this is the best solution I've seen so far.

[noxdafox]

Hello,

I created issue #16 to reflect your issue. Please feel free to provide feedback regarding the configuration format proposal.