[BUG] Opensearch 1.3.3 plugins - missing indexes breaks functionality

[DefenceLogicAdm]

What is the bug? Opensearch-Dashboards 1.3.3 - missing indexes .opendistro-anomaly-detectors / .opendistro-alerting-config breaks the alerting and anomaly detection functions.

How can one reproduce the bug? Steps to reproduce the behavior:

1. start opensearch-dashboards with /usr/share/opensearch-dashboards/bin/opensearch-dashboard from command line as normal user
2. Log into admin UI
3. Click on Alerting in left hand window
4. Click on Anomaly Detection
5. Observer errors in logs

What is the expected behavior? Errors should not occur, indexes should have been already recreated.

What is your host/environment?

• Centos 7
• Opensearch 1.3.3 / Opensearch-Dashboards 1.3.3
• Plugins alertingDashboards@1.3.3.0 anomalyDetectionDashboards@1.3.3.0 ganttChartDashboards@1.3.3.0 indexManagementDashboards@1.3.3.0 observabilityDashboards@1.3.3.0 queryWorkbenchDashboards@1.3.3.0 reportsDashboards@1.3.3.0 securityDashboards@1.3.3.0

Observed Log Outpout

[cyberkryption@opensearch ~]$ /usr/share/opensearch-dashboards/bin/opensearch-dashboards
  log   [13:49:36.320] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [13:49:36.430] [info][plugins-system] Setting up [44] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,alertingDashboards,share,opensearchUiShared,embeddable,legacyExport,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,ganttChartDashboards,reportsDashboards,securityDashboards,anomalyDetectionDashboards,indexManagementDashboards,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,observabilityDashboards,bfetch]
  log   [13:49:36.722] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations...
  log   [13:49:36.978] [info][savedobjects-service] Starting saved objects migrations
  log   [13:49:37.037] [info][plugins-system] Starting [44] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,alertingDashboards,share,opensearchUiShared,embeddable,legacyExport,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,ganttChartDashboards,reportsDashboards,securityDashboards,anomalyDetectionDashboards,indexManagementDashboards,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,savedObjectsManagement,observabilityDashboards,bfetch]
  log   [13:49:37.404] [info][listening] Server running at http://opensearch.cyberkryption.local:5601
  log   [13:49:37.513] [info][server][OpenSearchDashboards][http] http server running at http://opensearch.cyberkryption.local:5601
Anomaly detector - Unable to search detectors { Error: [index_not_found_exception] no such index [.opendistro-anomaly-detectors], with { index=".opendistro-anomaly-detectors" & resource.id=".opendistro-anomaly-detectors" & resource.type="index_or_alias" & index_uuid="_na_" }
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 404,
  displayName: 'NotFound',
  message:
   '[index_not_found_exception] no such index [.opendistro-anomaly-detectors], with { index=".opendistro-anomaly-detectors" & resource.id=".opendistro-anomaly-detectors" & resource.type="index_or_alias" & index_uuid="_na_" }',
  path: '/_plugins/_anomaly_detection/detectors/_search',
  query: {},
  body:
   { error:
      { root_cause: [Array],
        type: 'index_not_found_exception',
        reason: 'no such index [.opendistro-anomaly-detectors]',
        index: '.opendistro-anomaly-detectors',
        'resource.id': '.opendistro-anomaly-detectors',
        'resource.type': 'index_or_alias',
        index_uuid: '_na_' },
     status: 404 },
  statusCode: 404,
  response:
   '{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.opendistro-anomaly-detectors]","index":".opendistro-anomaly-detectors","resource.id":".opendistro-anomaly-detectors","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [.opendistro-anomaly-detectors]","index":".opendistro-anomaly-detectors","resource.id":".opendistro-anomaly-detectors","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}',
  toString: [Function],
  toJSON: [Function] }
Alerting - MonitorService - searchMonitor: { Error: [alerting_exception] Configured indices are not found: [.opendistro-alerting-config]
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 404,
  displayName: 'NotFound',
  message:
   '[alerting_exception] Configured indices are not found: [.opendistro-alerting-config]',
  path: '/_plugins/_alerting/monitors/_search',
  query: {},
  body:
   { error:
      { root_cause: [Array],
        type: 'alerting_exception',
        reason:
         'Configured indices are not found: [.opendistro-alerting-config]',
        caused_by: [Object] },
     status: 404 },
  statusCode: 404,
  response:
   '{"error":{"root_cause":[{"type":"alerting_exception","reason":"Configured indices are not found: [.opendistro-alerting-config]"}],"type":"alerting_exception","reason":"Configured indices are not found: [.opendistro-alerting-config]","caused_by":{"type":"exception","reason":"org.opensearch.index.IndexNotFoundException: no such index [.opendistro-alerting-config]"}},"status":404}',
  toString: [Function],
  toJSON: [Function] }

Do you have any additional context? Tried creating indexes with curl manually but index is missing data structures and results in a 500 internal error 0 unable to search indexes

How to resolve Provide documentation on the .opendistro-anomaly-detectors / .opendistro-alerting-config index structure ideally this would be two json files that could be used with curlor appropriate commands.

[llermaly]

same in 2.4.0 and document based alerts

[ohltyler]

Hi @DefenceLogicAdm @llermaly thanks for the issue. This is actually expected behavior since these are system indices that are not supposed to be directly accessible to users (note they are prefixed with .). These indices are created automatically by the plugin when plugin resources, like anomaly detectors or alerting monitors, are created. But by default they are not created on initial cluster bootstrap since they would be unnecessary if the plugin never gets used.

[kaituo]

@ohltyler reopened so that we can catch index_not_found_exception and suppress the message on the backend.