Authorization token valid fail

s4kharitonov commented 5 years ago

Hello, I noticed that several messages not received to zoom. At one moment, 4 warnings worked for me, 3 messages came in zoom and one was lost. After a thorough check of the logs, I see that this happens periodically.

docker logs:

got 1 alerts, start processing.
got 1 alerts, start processing.
got 1 alerts, start processing.
got 1 alerts, start processing.
handle response of [req-6d77e406-01b8-4155-9031-4b15170d7236] - 200
 {"message":"authorization token valid fail.","code":7010}
handle response of [req-228f7d70-080d-4436-97de-3b1387f1af15] - 200
 "OK"
handle response of [req-e17e4294-6361-4ea4-b95e-c5db8f107974] - 200
 "OK"
handle response of [req-c69fdf0b-c237-4dd2-811a-4b18893a23fc] - 200
 "OK"

SERVER_VERSION=1.0.12

Also it seems to me that link "Click here to view detail metrics" work incorrectly. After click on it, I got this error: Error executing query: invalid parameter 'query' and full query look:

haproxy_backend_up %3D%3D 0

Must be

haproxy_backend_up == 0

Could you check it when you have spare time. Thanks in advance.

Code2Life commented 5 years ago

I didn't encounter the first issue. I'm afraid it was something related to incoming webhook bot. could you plz try to build another imcoming webook? if it still happens, you could relay your alertmanager alerts to http://webhook.site/ and paste request detail for trouble shooting.

As for second issue, could you paste the link address after clicking? I use prometheus version of 2.7.2 , it's fine, what's your prometheus version?

s4kharitonov commented 5 years ago

prometheus version

prometheus, version 2.12.0 (branch: HEAD, revision: 43acd0e2e93f9f70c49b2267efa0124f1e759e86)
  build user:       root@7a9dbdbe0cc7
  build date:       20190818-13:53:16
  go version:       go1.12.8

alertmanager version

alertmanager, version 0.19.0 (branch: HEAD, revision: 7aa5d19fea3f58e3d27dbdeb0f2883037168914a)
  build user:       root@587d0268f963
  build date:       20190903-15:01:40
  go version:       go1.12.8

link after clicked in zoom chat

http://domain:9090/graph?g0.expr=haproxy_backend_up+%253D%253D+0&g0.tab=1

link after clicked in alertmanager source button

http://domain:9090/graph?g0.range_input=1h&g0.expr=haproxy_backend_up%20%3D%3D%200&g0.tab=1

Code2Life commented 5 years ago

"%253D%253D" is not valid encoded URL, could you plz provide raw alert request detail through webhook.site so that i could reproduce the issue?

s4kharitonov commented 5 years ago

If send to webhook.site link work correct

{
  "head": {
    "text": "Alert Status: FIRING",
    "style": {
      "color": "#FF0000",
      "bold": true,
      "italic": false
    }
  },
  "body": [
    {
      "type": "message",
      "text": "Instance: ip:22,Job: SSH",
      "style": {
        "color": "#666",
        "bold": false,
        "italic": false
      }
    },
    {
      "type": "message",
      "text": "[1] 2019-10-28T10:51:09.439Z SSH Critical",
      "style": {
        "color": "#222222",
        "bold": true,
        "italic": false
      }
    },
    {
      "type": "message",
      "text": "Description: ip:22 | TCP: Failed SSH check",
      "style": {
        "color": "#443333",
        "bold": false,
        "italic": false
      }
    },
    {
      "type": "message",
      "text": "Click here to view detail metrics",
      "link": "http://domain:9090/graph?g0.expr=probe_success%7Bjob%3D%22SSH%22%7D+%3D%3D+0&g0.tab=1"
    },
    {
      "type": "message",
      "text": "Click here to see all alerts",
      "link": "http://domain:9093"
    },
    {
      "type": "message",
      "text": "Click here to open grafana",
      "link": "http://domain:3000"
    }
  ]
}

s4kharitonov commented 5 years ago

webhook.site

http://domain:9090/graph?g0.expr=probe_success%7Bjob%3D%22SSH%22%7D+%3D%3D+0&g0.tab=1

zoom

http://domain:9090/graph?g0.expr=probe_success%257Bjob%253D%2522SSH%2522%257D+%253D%253D+0&g0.tab=1

alertmanager source

http://domain:9090/graph?g0.expr=probe_success%7Bjob%3D%22SSH%22%7D+%3D%3D+0&g0.tab=1

s4kharitonov commented 5 years ago

Sent test alert with correct url, after click on link url is incorrect

curl -X POST http://domain:9093/api/v1/alerts \
  -H 'Content-Type: application/json' \
  -d '[{
"labels": {
"alertname": "Test",
"service": "Service",
"severity": "critical",
"instance": "10.10.10.12"
},
"annotations": {
  "summary": "Test service is down!"
},
"generatorURL": "http://domain:9090/graph?g0.expr=probe_success%7Bjob%3D%22SSH%22%7D+%3D%3D+0&g0.tab=1"
}]'

After clicked

http://domain:9090/graph?g0.expr=probe_success%257Bjob%253D%2522SSH%2522%257D+%253D%253D+0&g0.tab=1

Also tried change channel, generated new zoom webhook url and token, no luck, still the same problem

Code2Life commented 5 years ago

It's wired. I will try to reproduce locally...

s4kharitonov commented 5 years ago

@Code2Life Any update for this? Can you add decodeURI for alert.generatorURL? It should solve problem with link

Code2Life commented 5 years ago

@s4kharitonov I'll add this tomorrow

Code2Life commented 5 years ago

I coundn't reproduce the issue with same prometheus/alertmanager version. Adding decodeURI causes 400 Bad Request from Zoom side. What's your ZOOM client version? Could you please upgrade your Zoom Client to latest version and retry ?

s4kharitonov commented 5 years ago

I tried to update zoom client, it didn't solve the problem. In any case, I no longer have any ideas why this is not working correctly for me. In this case, I simply will not display this link, thanks for the help.

Code2Life commented 5 years ago

You are welcome, and thanks for discovering issues. but this issue is wired and I could not reproduce. I'll contact Zoom Integration team to make an alertmanager native integration app, without extra deployment. It may cause some time, will update this repo when they publish the app in marketplace

Code2Life / nodess-apps

Authorization token valid fail #2