search

jjethwa / icinga2

GNU General Public License v3.0
222 stars 188 forks source link

Question: how best to integrate with PagerDuty? #180

Open sdanieru opened 5 years ago

sdanieru commented 5 years ago

Greetings! I'd like to integrate your icinga2 image w/ PagerDuty. What's the best way to do this? Per this page: https://www.pagerduty.com/docs/guides/icinga2-integration-guide/ I've made a Dockerfile to wrap your image w/ the additional changes mentioned in the integration guide, but I'm having difficulty getting both the cron and /opt/run to run at the same time. Here's what I have so far:

FROM jordan/icinga2:latest

RUN apt-get update \
  && apt-get install -y libwww-perl libcrypt-ssleay-perl libsys-syslog-perl cron

COPY --chown=nagios:root conf/pagerduty-icinga2.conf /etc/icinga2/conf.d/pagerduty-icinga2.conf
COPY --chown=nagios:root conf/templates.conf /etc/icinga2/conf.d/templates.conf
COPY --chown=nagios:root bin/pagerduty_icinga.pl /usr/local/bin/pagerduty_icinga.pl
COPY hello-cron /etc/cron.d/hello-cron

CMD ["&& cron -f"]
-

since ENTRYPOINT is used in the base img, CMD ought to append '&& cron' - but it looks like cron hasn't started, as it's not writing to the log. As I'm thinking about this more, perhaps it's never getting to '&& cron' since /opt/run is still running...

Not sure if I'm on the right track here, or if there's a better way. Any help appreciated!

jjethwa commented 5 years ago

Hi @sdanieru

I would probably use supervisord (it's already installed and running icinga, mysql, etc) to run the task you need or to run the cron daemon instead. Hope that helps :)

sdanieru commented 5 years ago

that is helpful, thanks for the quick response, appreciate it! If I can get it working I'll leave details here.

jjethwa commented 5 years ago

Sounds good, thanks @sdanieru Let me know if you need any help 😄

sdanieru commented 5 years ago

so I've successfully wrapped your image w/ the specifics that the pagerduty integration needed. When I started the container it had some built-in checks that immediately went critical (e.g. icinga2:apt) and it did send the alerts to pagerduty, so it does work, but I am somewhat confused, as the pre-installed service checks don't show up in 'Icinga Director' > Services and it's unclear to me how to 'hook' into the pagerduty notifications. The services I've setup, I see them correctly going from OK to Critical when I bring the monitored service down, but they never send notifications. In the service editor, there's no way to add a notification. One approach I tried was to add the service to a group named 'prod' and then in the notification under 'assign where' I added ["service.groups"] contains prod but this didn't work. Another item I noticed, is in the built-in alerts, they list several contacts and contactGroup, which included the PagerDuty Notifier, but I don't see these contacts or contactGroup anywhere in the Icinga Director.

jjethwa commented 5 years ago

Hi @sdanieru

That's great news about the integration. I'm not a Director expert , but I believe that it does not show objects that are pre-defined (/etc/icinga2/conf.d/ or /usr/share/icinga2). Just to make sure, you set up:

  1. Notification template
  2. NotificationCommand
  3. Notification
  4. UserGroup
  5. User

That's the minimum setup needed IMO. Take a look at my Slack notification repo: https://github.com/jjethwa/icinga2-slack-notification for an example