Use a notifier plugin in GoCD

[timmc-edx]

A/C

• [ ] (timebox 1-2 days) establish the difficulty in using the GoCD email notifier or modifying another existing plugin
• [ ] If it's fast, update GoCD to use the plugin instead of the existing email notifications that are built into GoCD
• [ ] If it's not fast, discuss with the team whether or not to create a ticket for implementing or discarding the idea

GoCD's built-in notifications system occasionally stops sending notifications, requiring manual intervention after server restart. We've also tried adding explicit failure/success calls to Opsgenie, but it's verbose and has various downsides.

Another option is to add a notifier plugin.

Implementation notes:

• GoCD has a Notification Plugin API: https://plugin-api.gocd.org/current/notifications/
• There's an existing plugin to send emails for stage state changes: https://github.com/gocd-contrib/email-notifier -- we may be able to just us this, off-the-shelf, alongside some new Opsgenie rules.
• If we want to modify or create a plugin, they're written in Java and the gocd-contrib GitHub org has several additional examples we could start from.

There are other APIs we can call; we may be able to send the output logs to Opsgenie by using the Artifacts API to retrieve cruise-output/console.log.

[robrap]

@timmc-edx: This has bubbled near the top of the on-call backlog, but doesn't strike me as more important than other work, including main focus work. What are your thoughts? Do you want to discuss as a team?

[timmc-edx]

I think it would be pretty quick to try the existing email notifier plugin, and likely worth the effort.

[timmc-edx]

Status: We're overall reasonably happy with the behavior of the current approach, in which we have calls to Opsgenie at the end of each job and in a separate stage at the end of each pipeline. However, it's a real bear to maintain—changing the pipeline (adding and removing jobs and stages, reordering, etc.) is now error-prone and involves a lot of extra code. So we'd likely still benefit from a proper notifier plugin, but more so if we're about to embark on changes to the edxapp pipelines.