Default _list/shards API call (curl localhost:9200/_list/shards) fails with index_closed_exception if an Opensearch cluster has one or more closed indices.
The issue occurs due to the way pagination is implemented in the _list APIs. After processing the clusterState response, concrete list of indices is passed to other TransportActions down the line. For _list/shards API, all the concrete indices processed and fetched from PaginationStrategy are directly supplied to TransportIndicesStatsAction.
By default, TransportIndicesStatsAction uses IndicesOptions.strictExpandOpenAndForbidClosed() which results in index_closed_exception if any of of the concrete index is closed.
The reason this same thing doesn't fail in _list/indices is due to the fact that, a TransportGetSettingsAction call is first made with whatever indices are specified in the URL (not the concrete indices from the strategy) [Code Ref] and all the other TransportAction calls post that are made with IndicesOptions.lenientExpandHidden();[Code Ref].
Fix:
Filter out closed indices as part of ShardPaginationStrategy (accept indicesOptions as part of strategy) and let TransportIndicesStatsAction process only the open indices.
Related component
Cluster Manager
To Reproduce
Have one or more closed indices in a cluster.
Make _list/shards (curl localhost:9200/_list/shards) call, which would fail with index_closed_exception.
Expected behavior
_cat/shards API, displays the closed shards without stats for an unfiltered query. The _list/shards API similarly, should not fail with exception and instead just filter out the response.
Describe the bug
Default
_list/shardsAPI call (curl localhost:9200/_list/shards) fails withindex_closed_exceptionif an Opensearch cluster has one or more closed indices.The issue occurs due to the way pagination is implemented in the _list APIs. After processing the clusterState response, concrete list of indices is passed to other TransportActions down the line. For
_list/shardsAPI, all the concrete indices processed and fetched fromPaginationStrategyare directly supplied toTransportIndicesStatsAction. By default,TransportIndicesStatsActionusesIndicesOptions.strictExpandOpenAndForbidClosed()which results inindex_closed_exceptionif any of of the concrete index is closed.The reason this same thing doesn't fail in
_list/indicesis due to the fact that, aTransportGetSettingsActioncall is first made with whatever indices are specified in the URL (not the concrete indices from the strategy) [Code Ref] and all the other TransportAction calls post that are made withIndicesOptions.lenientExpandHidden();[Code Ref].Fix:
Filter out closed indices as part of
ShardPaginationStrategy(acceptindicesOptionsas part of strategy) and letTransportIndicesStatsActionprocess only the open indices.Related component
Cluster Manager
To Reproduce
curl localhost:9200/_list/shards) call, which would fail withindex_closed_exception.Expected behavior
_cat/shardsAPI, displays the closed shards without stats for an unfiltered query. The_list/shardsAPI similarly, should not fail with exception and instead just filter out the response.Additional Details
No response