oxyno-zeta / prometheus-cachethq

Prometheus Alertmanager wehbooks to CachetHQ
https://oxyno-zeta.github.io/prometheus-cachethq/
Apache License 2.0
11 stars 3 forks source link

Filter component using CachetHQ tags (or groups) #4

Closed renebarbosafl closed 4 years ago

renebarbosafl commented 4 years ago

We have several components with the same name (for example "API") in multiple groups. When we send an update to the component searching for the name, all of them are being updated.

It would be nice to find the components using tags.

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the label feature_request to this issue, with a confidence of 0.92. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

oxyno-zeta commented 4 years ago

Hello !

First of all, thanks for your good first issue. I must admit that I'm surprised that this project is used by someone else :) .

I'm facing a problem with your request: the golang client I'm using in this project haven't the support for tags (and seems not maintained unfortunately...).

So I can imagine two solutions here:

1)

But: How to manage the fact that multiple groups have the same set of tags ?

2)

   component:
      name: API
      group: FAKE1 # optional
      status: PARTIAL_OUTAGE

I must admit that I'm more in favour of the solution 2. Do you see any problems with that ?

renebarbosafl commented 4 years ago

Hello Alexandre!

Thanks for your prompt answer! Yeah we are trying to use your project with our monitoring system and so far it's promising. :)

About your proposal.. I think the solution #2 will work just fine, even without the support for tags. In our example:

Group: CustomerA Component: API

Group: CustomerB Component: API

So we just need to use the "group: CustomerA" and "name: API" in our component declaration and it will work! Nice!

oxyno-zeta commented 4 years ago

I've just released the 1.1.0 with this feature. Thanks for this.

Tell me if everything is ok.

Regards,

renebarbosafl commented 4 years ago

Thanks for the update and feature! Testing this right now with this config:

  - component:
      name: API
      group: Alagoas
      status: PERFORMANCE_ISSUES
    alerts:
      - name: ApiPerformance

We forced an alert and our component wasn't updated. Log from alertmanager:

level=error ts=2020-09-03T15:11:34.755Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="cachethq-receiver/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 404: http://prom-cachethq:8080/prometheus/webhook"

For some reason it's not finding the components using Group and name.

oxyno-zeta commented 4 years ago

Hello !

The configuration isn't valid. The key is "groupName" and not "group". Your key has been ignored. I've changed the key from the upper example. Why ? Because if in future we need to have a key called "groupID", it will be clear :wink:.

Hope this will work now.

PS: sorry for the short answer, I'm on my phone :smiley:

renebarbosafl commented 4 years ago

Awesome, worked like a charm! Thank you.

oxyno-zeta commented 4 years ago

@renebarbosafl Cool great to know :) .

Thanks for your feature.