Opsgenie integration with Kubesphere alertmanager

[KR411-prog]

I am trying to integrate Opsgenie with Kubesphere alertmanager..But facing issue on this. Looks like Notification manager does not support Opsgenie. Note: Prometheus Operator works with opsgenie_config.
Below are the steps I tried: Created API integration in Opsgenie and provided the API url and api key in Kubesphere Notification management Webhook configuration,

It is throwing authentication error, http error, code: 401, message: {"message":"Could not authenticate","took":0.001,"requestId":"d836ad60-8ad5-4ba7-a3a1-94fb358a10ee"}

I also tried to update the alertmanager secret as below, `kubectl edit secret alertmanager-main -n kubesphere-monitoring-system

    -   name: opsgenie_channel
        opsgenie_configs:
        -   api_key: c624xxxx-dxx-4xxf-9fxa-9c10xxxxx62
            api_url: https://api.xx.opsgenie.com
            responders:
            -   name: Team1
                type: team`

But it had no effect..

I find kubesphere/ks-installer/roles/ks-monitor/files/prometheus/prometheus-operator/prometheus-operator-0alertmanagerConfigCustomResourceDefinition.yaml file shows opsgenie_configs so I thought it is still supported. But looks like it is not.. Is there a way to integrate opsgenie with Kubesphere?

[KR411-prog]

So it turns out to be the request message format sent by Kubesphere to Opsgenie api call.. Its not accepting that format We didnt have this issue with Prometheus operator alertmanager as it seems to already send proper message format

This is the format expected by Opsgenie https://docs.opsgenie.com/docs/alert-api#create-alert

I reached out to Opsgenie support and they mentioned they can also open up a feature request to support Kubesphere requests.. "Can you provide use with an explanation of how you would imagine this integration to work with OpsGenie? What’s the use case? Some examples below to help you think about it: I need Alarms in tool X to create alerts in OpsGenie. I’d also like OpsGenie to acknowledge the alert when I set the problem to “in progress” in X. I need to forward OpsGenie alerts to Y as new tickets. Plus, I need to be able to change ticket status when I ack/close the alert. I need to have two-way synchronization between my alerts in tool P and OpsGenie. So, if I ack the alert in P it should auto-ack the OpsGenie alert, and vice versa. It would be nice to execute my subroutines in tool Z from the OpsGenie alert’s custom actions. 2) Is the tool’s network open to incoming connections from the outside world? Or is it used in an internal network?3) If the tool is not Saas and on-premise instead, which Version of the tool are they using? Which system distribution (linux x64, ubuntu, windows, etc.) does it run on?4) Are you willing to test the integration for us when the initial development is complete?"

[FeynmanZhou]

Hi team @kubesphere/sig-observability , @benjaminhuo @VioZhang @wanjunlei ,

PTAL at this feature request, thanks. Do we need to migrate this issue to the Notification Manager repo? It seems OpsGine is also quite popular for alerting and notification management. We might need to investigate how to support OpsGine with the KubeSphere Observability system. The community could help to develop it if this proposal makes sense.

[wanjunlei]

@KR411-prog KubeSphere uses Notification Manager to send notifications, not alertmanager. Now the Notification Manager does not support OpsGine. Welcome to submit pr to support OpsGine。

Alertmanager support to send alerts to OpsGine, you can refer to opsgenie config for more information. You can check the alertmanager log for errors if it had no effect.

[benjaminhuo]

@KR411-prog The Prometheus operator defines opsgenie CRDs, you can use these to create opsgenie CR https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#opsgenieconfig
And yes , welcome to add opsgenie support to Notification Manager