opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.76k stars 1.81k forks source link

[BUG] [Remote State] Timeout of remote state download during boostrap #16055

Open soosinha opened 1 month ago

soosinha commented 1 month ago

Describe the bug

When remote cluster state is enabled, we upload the cluster state to remote store. When new cluster manager nodes are added, these download the latest remote state during bootstrap. But this download gets timed out where there are large number of indices (>10k).

Related component

Cluster Manager

To Reproduce

  1. Create a cluster
  2. Add 10000 indices in the cluster
  3. Add a new cluster manager node
  4. Observe whether the new cluster manager is not be able to join the cluster.

Expected behavior

Download of remote state should not get timed out. We should optimize the download to have faster download and increase the default timeout as well.

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

Additional context Add any other context about the problem here.

dblock commented 3 weeks ago

[Catch All Triage - 1, 2, 3, 4]