Bugs with `index.number_of_routing_shards` Setting

Rassyan commented 3 weeks ago

Elasticsearch Version

8.14.0

Installed Plugins

No response

Java Version

bundled

OS Version

N/A

Problem Description

Description

There are critical issues with the index.number_of_routing_shards setting in Elasticsearch:

Default Value Mismatch: The default value of index.number_of_routing_shards is set to index.number_of_shards, which does not align with the actual computed value of routing_num_shards.
Post-Close Modifiability: index.number_of_routing_shards can be modified after the index is closed, but routing_num_shards remains unchanged, leading to inconsistencies.
Retrieval Discrepancies: When index.number_of_routing_shards is explicitly set, it is not retrievable via the _settings API and can only be accessed through _cluster/state/metadata.

These issues cause confusion and misconfiguration, as the actual routing behavior is governed by routing_num_shards, which is not directly accessible or modifiable in a straightforward manner.

Summary

To address these issues:

Documentation: Clarify the default value calculation for index.number_of_routing_shards and its relationship with routing_num_shards.
API Consistency: Ensure the _settings API returns the actual value of index.number_of_routing_shards and aligns with routing_num_shards.
Parameter Modification: Prohibit modification of index.number_of_routing_shards after index creation or clearly document its immutability.

These changes will help users better understand and configure their indices, avoiding potential pitfalls and confusion.

Steps to Reproduce

Step 1: Default Value Mismatch

DELETE test

PUT test
{
  "settings": {
    "index.number_of_shards": 3
  }
}

GET test/_settings?include_defaults&filter_path=**.*routing*shards
---
{
  "test": {
    "defaults": {
      "index": {
        "number_of_routing_shards": "3"
      }
    }
  }
}

GET /_cluster/state/metadata/test?filter_path=**.*routing*shards
---
{
  "metadata": {
    "indices": {
      "test": {
        "routing_num_shards": 768
      }
    }
  }
}

Step 2: Post-Close Modifiability

POST test/_close

PUT test/_settings
{
  "settings": {
    "index.number_of_routing_shards": 6
  }
}
---
{
  "acknowledged": true
}

POST test/_open

GET test/_settings?include_defaults&filter_path=**.*routing*shards
---
{
  "test": {
    "settings": {
      "index": {
        "number_of_routing_shards": "6"
      }
    }
  }
}

GET /_cluster/state/metadata/test?filter_path=**.*routing*shards
---
{
  "metadata": {
    "indices": {
      "test": {
        "routing_num_shards": 768,
        "settings": {
          "index": {
            "number_of_routing_shards": "6"
          }
        }
      }
    }
  }
}

Step 3: Retrieval Discrepancies

DELETE test1

PUT test1
{
  "settings": {
    "index.number_of_shards": 3,
    "index.number_of_routing_shards": 6
  }
}

GET test1/_settings?include_defaults&filter_path=**.*routing*shards
---
{
  "test1": {
    "defaults": {
      "index": {
        "number_of_routing_shards": "3"
      }
    }
  }
}

GET /_cluster/state/metadata/test1?filter_path=**.*routing*shards
---
{
  "metadata": {
    "indices": {
      "test1": {
        "routing_num_shards": 6
      }
    }
  }
}

Logs (if relevant)

No response

elasticsearchmachine commented 3 weeks ago

Pinging @elastic/es-distributed (Team:Distributed)

ywangd commented 2 weeks ago

I am re-labelling this to :Data Management/Indices APIs since both linked issues are with that label. Please feel free to reroute if it should be looked into by the distributed team.

elasticsearchmachine commented 2 weeks ago

Pinging @elastic/es-data-management (Team:Data Management)

elastic / elasticsearch