Not sending alerts to Slack

[Murryy79]

Hello: I can see alerts are triggered in AlertManager. State is FIRING. But I get no notification in Slack.

my manifest.yml configs to send alerts to slack:

Note: I am able to successfully send test alerts via below:

but I had to first configure company proxy like this export http_proxy='\''http://proxy.df.com:8080'\'' export https_proxy='\''http://proxy.df.com:8080'\'''

This test below tells {"status":"success"} but nothing is sending to slack

I wonder no alerts are sent to slack because of PCF/BOSH behind company proxy ? if yes, how do i configure proxy in manifest.yml configs? or could be something else . i am clueless

any assistance is greatly appreciated...

Thanks Murryy

[Murryy79]

i was digging in some logs and saw these. thought will post here time="2017-08-06T11:53:08Z" level=error msg="Notify for 9 alerts failed: context deadline exceeded" source="dispatch.go:261" time="2017-08-06T11:54:43Z" level=error msg="Error on notify: context deadline exceeded" source="notify.go:283" time="2017-08-06T11:54:43Z" level=error msg="Notify for 1 alerts failed: context deadline exceeded" source="dispatch.go:261" time="2017-08-06T11:55:27Z" level=error msg="Error on notify: context deadline exceeded" source="notify.go:283" time="2017-08-06T11:55:27Z" level=error msg="Notify for 1 alerts failed: context deadline exceeded" source="dispatch.go:261"

[mkuratczyk]

Hi. Proxy could definitely be the culprit here. Proxy can be set using Alertmanager job properties: https://github.com/cloudfoundry-community/prometheus-boshrelease/blob/8d9e0961bc578ee0a20823ba01a6151f5f01d5de/jobs/alertmanager/spec#L113-L118. So you can try something like this in your manifest:

jobs:
- name: alertmanager
  properties:
    env:
      http_proxy: http://proxy.df.com:8080
      https_proxy: http://proxy.df.com:8080
    alertmanager:
      (no changes here)

[Murryy79]

@mkuratczyk I tried to set proxy like suggested above in manifest but nothing showed up in alertmanager.yml. Then i manually added following in alertmanager.yml env: http_proxy: http://proxy.df.com:8080 https_proxy: http://proxy.df.com:8080

alertmanager process will not start and throw following error time="2017-08-15T23:52:43Z" level=error msg="Loading configuration file failed: unknown fields in config: env" file="/var/vcap/jobs/alertmanager/config/alertmanager.yml" source="main.go:209"

[mkuratczyk]

Hey. I don't know if that's just formatting after pasting here or your actual YAML file formatting but the error suggests the latter (an incorrectly formatted YAML config file). Would you mind attaching your alertmanager.yml and/or your (redacted) manifest?

[Murryy79]

I tried same steps in another PCF environment and slack notifications started working just fine. Thank you for your assistance @mkuratczyk I think there's some network connectivity issue in sandbox foundation where i was testing earlier.

additionally, I am tasked to convert the alerts into ServiceNow[A tool for ticket Incident tickets.] I know if we can send email alerts to email address, ServiceNow can convert into incident tickets.

I got email alerting working as well.

my question is how do I setup multiple receivers in manifest.yml eg for both Slack and email. Add a custom subject or line to body of the email

my manifest.yml that didnt' work for multiple receivers. what am i doing wrong?

[mkuratczyk]

I haven't had to configure multiple receivers myself so not sure. Your configuration looks similar to https://www.robustperception.io/sending-alert-notifications-to-multiple-destinations/ but perhaps the job template distorts the config in this case? Please attach your alertmanager.yml as it appears on the VM. Also, check Alertmanager's logs (there could be parsing errors or something).

[Murryy79]

attaching alertmanager.yml and logs.

The global and env section in alertmanager.yml is manually added by me. I like those configurations to come from manifest.yml so I can just run my concourse pipeline and don't have to touch anything inside VM configuration files.

[mkuratczyk]

I don't see any attachments. Did you forget to include them?

[Murryy79]

@mkuratczyk Earlier i tried to attach but failed everytime because here .yml or .log extentions are rejected. so i am renaming to files with .txt

I got most of the things working by following examples. Here's my real issues.

1. I can get either slack alerts or email alerts working. but not both.
2. Another issue is in email alerts I don't know it's coming from which PCF foundations[we have dev and prod PCF] . So I need to add a customized subject line. my alertmanager.yml , alertmanager.stderr.log,manifest.yml attached I don't know what I am doing wrong in setting up the receivers

[mkuratczyk]

Hi. I'd recommend just playing with the alertmanager.yml directly on the VM to take any potential bosh-release issues out of the picture - just use Alertmaanger's docs/forums to get that working and then try to achieve the same config file through the BOSH manifest. As I said, I haven't had to send alerts to multiple destinations and looking at your alertmanager.yml it looks pretty good to me.

As for multiple environments - first of all, do you have a single Alertmanager for both environments? Because if not then you can just set a static Subject header per env. If you have a single Alertmanager then you should be able to use Alertmanager's templating based on a label (https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/).

[mkuratczyk]

I hope you managed to solve this problem.

[Murryy79]

Thanks @mkuratczyk I was able to to send alerts to multiple receivers. Here's my working example in case someone's looking for it