opensearch-project / OpenSearch-Dashboards

📊 Open source visualization dashboards for OpenSearch.
https://opensearch.org/docs/latest/dashboards/index/
Apache License 2.0
1.68k stars 882 forks source link

Creating index-patterns causes search threadPool to be exhausted #8596

Open kkewwei opened 4 days ago

kkewwei commented 4 days ago

Is your feature request related to a problem? Please describe.

  1. when I set the ccs in the cluster settings(the address is pointed to itself to reproduce)
PUT _cluster/settings
{
  "persistent": {
      "cluster": {
         "remote": {
            "ccs1": {
               "mode": "proxy",
               "proxy_address": "127.0.0.1:9300"
            }
         }
      }
  }
}

and create a new index-patterns in the dashboard, I find that search thread pool is immediately exhausted in the target cluster:

image

And search slow log is as follows:

image
POST *:*/_search
{
  "size": 0,
  "timeout": "30000ms",
  "query": {
    "match_all": {
      "boost": 1
    }
  },
  "track_total_hits": 2147483647,
  "aggregations": {
    "indices": {
      "terms": {
        "field": "_index",
        "size": 200,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "order": [
          {
            "_count": "desc"
          },
          {
            "_key": "asc"
          }
        ]
      }
    }
  }
}
  1. As the total_shards=35720, which equals the total number shards of the cluster, we know that all the indices are aggregated by the query.

  2. We know that the query is from the "Index_Patterns". image

It seems that we send the query to get the indices list of target cluster, when the target cluster is very large, this query will become very dangerous.

Describe the solution you'd like If we should add the api in the opensearch code core to get indices list of the ccs target cluster?

HenryTheSir commented 3 days ago

Same issue: https://github.com/opensearch-project/index-management-dashboards-plugin/issues/1178