[BUG] Dashboard Alerting plugins fails to retrieve available notification channels

[waza-ari]

What is the bug?

When trying to configure an action for a monitor, OSD fails to retrieve the available notification configs with the following HTTP message:

GET https://<osd-url>/api/notifications/get_configs?from_index=0&max_items=5000&sort_field=name&sort_order=asc

Response:

{"statusCode":400,"error":"Bad Request","message":"[request query.config_type]: expected at least one defined value but got [undefined]"}

This prevents the user from configuring an action, as the dashboard will show that there are no notification channels available:

There's at least one other user affected by this issue.

How can one reproduce the bug? Steps to reproduce the behavior:

At least in my configuration, the following steps show the error:

1. Go to 'Notifications' > 'Channels' and create a notification channel. I've been using Slack, the same happens with eMail channels though.
2. Click on 'Alerting' > 'Monitors' and create a monitor. In my case, I'm using a per document monitor, simplest possible configuration
3. Add a trigger and try to add an action
4. See error

What is the expected behavior? I'm expecting to see the available monitors.

What is your host/environment?

The Ubuntu 22.04 systems are the host systems. OS and OSD is running on a Kubernetes cluster using the official Helm chart.

• OS: Ubuntu 22.04
• Version v2.14.0
• Plugins:

opensearch-cluster-master-0 opensearch-alerting                  2.14.0.0
opensearch-cluster-master-0 opensearch-anomaly-detection         2.14.0.0
opensearch-cluster-master-0 opensearch-asynchronous-search       2.14.0.0
opensearch-cluster-master-0 opensearch-cross-cluster-replication 2.14.0.0
opensearch-cluster-master-0 opensearch-custom-codecs             2.14.0.0
opensearch-cluster-master-0 opensearch-flow-framework            2.14.0.0
opensearch-cluster-master-0 opensearch-geospatial                2.14.0.0
opensearch-cluster-master-0 opensearch-index-management          2.14.0.0
opensearch-cluster-master-0 opensearch-job-scheduler             2.14.0.0
opensearch-cluster-master-0 opensearch-knn                       2.14.0.0
opensearch-cluster-master-0 opensearch-ml                        2.14.0.0
opensearch-cluster-master-0 opensearch-neural-search             2.14.0.0
opensearch-cluster-master-0 opensearch-notifications             2.14.0.0
opensearch-cluster-master-0 opensearch-notifications-core        2.14.0.0
opensearch-cluster-master-0 opensearch-observability             2.14.0.0
opensearch-cluster-master-0 opensearch-performance-analyzer      2.14.0.0
opensearch-cluster-master-0 opensearch-reports-scheduler         2.14.0.0
opensearch-cluster-master-0 opensearch-security                  2.14.0.0
opensearch-cluster-master-0 opensearch-security-analytics        2.14.0.0
opensearch-cluster-master-0 opensearch-skills                    2.14.0.0
opensearch-cluster-master-0 opensearch-sql                       2.14.0.0

Do you have any additional context?

Configured Channels:

GET /_plugins/_notifications/channels
{
  "start_index": 0,
  "total_hits": 1,
  "total_hit_relation": "eq",
  "channel_list": [
    {
      "config_id": "mksp-slack-id",
      "name": "Makerspace Slack",
      "description": "Makerspace OpenSearch Slack",
      "config_type": "slack",
      "is_enabled": true
    }
  ]
}

Configuration details:

GET _plugins/_notifications/configs/mksp-slack-id
{
  "start_index": 0,
  "total_hits": 1,
  "total_hit_relation": "eq",
  "config_list": [
    {
      "config_id": "mksp-slack-id",
      "last_updated_time_ms": 1717343062308,
      "created_time_ms": 1717343062308,
      "config": {
        "name": "Makerspace Slack",
        "description": "Makerspace OpenSearch Slack",
        "config_type": "slack",
        "is_enabled": true,
        "slack": {
          "url": "https://hooks.slack.com/services/xxx"
        }
      }
    }
  ]
}

Monitor Details

GET _plugins/_alerting/monitors/xMPu148BSfAVsH_Q7Msq
{
  "_id": "xMPu148BSfAVsH_Q7Msq",
  "_version": 5,
  "_seq_no": 463,
  "_primary_term": 1,
  "monitor": {
    "type": "monitor",
    "schema_version": 8,
    "name": "backend_errors",
    "monitor_type": "doc_level_monitor",
    "enabled": true,
    "enabled_time": 1717314694738,
    "schedule": {
      "period": {
        "interval": 1,
        "unit": "MINUTES"
      }
    },
    "inputs": [
      {
        "doc_level_input": {
          "description": "",
          "indices": [
            "app.backend-*"
          ],
          "queries": [
            {
              "id": "931d03de-c239-4fa5-8a42-96ccb59a6d4d",
              "name": "error_level",
              "fields": [],
              "query": "level:\"ERROR\"",
              "tags": [],
              "query_field_names": []
            }
          ]
        }
      }
    ],
    "triggers": [
      {
        "document_level_trigger": {
          "id": "rcQJ2I8BSfAVsH_QHCXK",
          "name": "backend_error",
          "severity": "1",
          "condition": {
            "script": {
              "source": "query[name=error_level]",
              "lang": "painless"
            }
          },
          "actions": []
        }
      }
    ],
    "last_update_time": 1717341945525,
    "data_sources": {
      "query_index": ".opensearch-alerting-queries",
      "findings_index": ".opensearch-alerting-finding-history-write",
      "findings_index_pattern": "<.opensearch-alerting-finding-history-{now/d}-1>",
      "alerts_index": ".opendistro-alerting-alerts",
      "alerts_history_index": ".opendistro-alerting-alert-history-write",
      "alerts_history_index_pattern": "<.opendistro-alerting-alert-history-{now/d}-1>",
      "query_index_mappings_by_type": {},
      "findings_enabled": false
    },
    "owner": "alerting"
  },
  "associated_workflows": []
}

[Hailong-am]

guess this backport PR will fix this issue https://github.com/opensearch-project/alerting-dashboards-plugin/pull/955, @AWSHurneyt

[waza-ari]

Im not very familiar with the code base, to me the PR indicates that there is an accompanying PR to it making other required changes.

Given that we see this issue in 2.14.0 (plus whatever version the other forum user is using, given its the latest tag 11d ago it might also be 2.14.0), does that mean that some parts got backported and others didn’t? I was unable to find evidence that would support this.

In dashboard-alerting PR #916 changes have been introduced which require the dashboard-notification PR 181 to be merged. Both PRs where merged, but only dashboard-notification PR 181 was backported to 2.x in PR 194 while the alerting changes from alerting PR #916 are not back ported yet (waiting for #955).

This means the backport in the notification repo happened before 2.14 was released, the backport in alerting is still open.

Note that this breaks alerting in the currently released version.

[tomuk5]

i'd like to add that im running 2.14 and also have the same problem as the OP, alerting is now fundamentally broken in the current release.

[waza-ari]

Looks like #966 was merged. Would it be possible to confirm it will address this issue and will be included in 2.15? Happy to close this issue then

[amsiglan]

Yes, #966 should fix the issue and we will be releasing 2.15 next week