[BUG] Dashboards Management / Index Pattern Page kills huge clusters by resolving all indices in remoteclusters

[HenryTheSir]

Describe the bug

A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. /app/management/opensearch-dashboards/indexPatterns (Dashboards Management -> Index Pattern)

Expected behavior A clear and concise description of what you expected to happen. Index Pattern Management is possible, without killing all clusters. Opening this page as administrator triggers a call to /internal/index-pattern-management/resolve_index/*:* which fires against all remote clusters and depending on the size of the cluster with s3 stored snapshots results in high cpu loads and long running search querys.

OpenSearch Version Please list the version of OpenSearch being used. 2.16 Dashboards Version Please list the version of OpenSearch Dashboards being used. 2.16 Plugins

Please list all plugins currently enabled. default Screenshots % If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Browser and version [e.g. 22]

Additional context

Add any other context about the problem here.

[LDrago27]

@opensearch-project/admin route it to ism . Since the issue is related to S3 snapshots we are re routing the ticket.

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[kkewwei]

When creating the index-pattern, the query is sent to the target cluster:

POST *:*/_search
{
  "size": 0,
  "timeout": "30000ms",
  "query": {
    "match_all": {
      "boost": 1
    }
  },
  "track_total_hits": 2147483647,
  "aggregations": {
    "indices": {
      "terms": {
        "field": "_index",
        "size": 200,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "order": [
          {
            "_count": "desc"
          },
          {
            "_key": "asc"
          }
        ]
      }
    }
  }
}

There seems no good way to obtain the index information in the target cluster, as we just want to get the count of indices documents in the target cluster, if we should just adjust the core code to support?

GET /{index}/_stats/{metric}   ->  GET {target_cluster}:*/_stats/docs