Closed einari closed 5 years ago
@einari I need to know which metrics we want push notifications on, and for which namespaces
Could we start with memory - as a test. Set it to 1GB right now - we'll be setting a memory quota for everything that is way less than that, but interesting to gain some more experience
1gb quota for what? "User namespace", node, pod or what were you thinking of?
Per Pod to start with.
Grafana's alert system is really flaky https://github.com/grafana/grafana/issues/10412 (at least at the version that we are currently using).
Effectively means that I cannot set up alerts as of now
This sits in developed; probably needs some clarification. The idea was to be able to push to mobile devices - using something like notification hubs in Azure: https://azure.microsoft.com/en-us/services/notification-hubs/
The goal is to have a variety of options for people to be able to subscribe to events. For our internal staff
@einari push notifications are apparently not that easy to set up.
However, Prometheus Alertmanager comes with "out-of-the-box" integrations for SMS and email. I think that setting the alertmanager up to send some specified alerts by sms and/or email would be our best bet for now.
We can do SMS - but from what I can see in the documentation, it shouldn't be hard to do Notification Hubs: https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-notification-hubs
https://docs.microsoft.com/en-us/azure/notification-hubs/
https://azure.microsoft.com/en-us/resources/samples/event-hubs-dotnet-user-notifications/
There should be a HTTPS endpoint to be able to do this. Can Prometheus post to a Web URL?
There is another alternative as well - Azure Alerts: https://azure.microsoft.com/en-us/resources/videos/introduction-to-webhooks-for-azure-alerts/
Alternatively, there are lots of out-of-the-box support for alerting. Including possible integrations with WebHook, there are support for services like: PagerDuty, Slack, VictorOps, OpsGenie, and more...
I think the easiest and best solution would be to use either PagerDuty or VictorOps for this, though they are paid services.
Or, perhaps Pushover for just mobile notifications
Or, we could use slack. Slack can give us mobile push notifications
We have Teams - so that will effectively do the same in that regards, don't want to introduce Slack to the mix. Let's investigate further. It should be very simple to use Notification Hubs. The idea was to then govern this from Azure AD with groups
We will park this issue for noe.
child of https://github.com/dolittle-platform/home/issues/71
We need to be able to get push notifications to our mobile phones based on different metrics. Grafana has an extensive alert system that should enable this