[Paginating ClusterManager Read APIs] Paginate _cat/indices API.

[gargharsh3134]

Is your feature request related to a problem? Please describe

As the number of indices grow in an opensearch cluster, the response size and the latency of the default _cat/indices API increases which makes it difficult not only for the client to consume such large responses, but also stresses out the cluster by making it accumulate stats across all the indices.

Thus, pagination will not only help in limiting the size of response for a single query but will also prevent the cluster from accumulating indices stats for all the indices. So, this issue tracks the approaches that can be used to paginate the response.

Describe the solution you'd like

Pagination key is index creation timestamps [RECOMMENDED]:

Overview: Each index has a creation timestamp stored in IndexMetadata which is part of Metadata object of ClusterState. These creation timestamps can act as sort/pagination key using which list of indices, sorted as per their respective creation timestamps, can be generated. The generated sorted list can then be used to prepare a list of indices to be sent as a response as per the page size.

User Experience:

Proposed New Query-> curl "localhost:9200/_cat/indices?format=json&pretty&nextToken=null&pageSize=1000"

Users can specify a nextToken as a query parameter in the rest request, starting with nextToken being null to get the first page.Such paginated queries will be responded with a list of indices stats, the number of indices in a page will be decided by the user provided parameter pageSize, and a nextToken which can then be used to fetch subsequent pages until nextToken in the response is again null (implying no further pages remaining to be fetched).

Proposed New JSON response ->

{
  "nextToken" : "MCQw",
  "indices" : [{
    "health" : "green",
    "status" : "open",
    "index" : "test-index1",
    "uuid" : "0TbPDgYBRkmifYvJGWYk9w",
    "pri" : "1",
    "rep" : "1",
    "docs.count" : "1",
    "docs.deleted" : "0",
    "store.size" : "7.8kb",
    "pri.store.size" : "3.9kb"
  },
  {
    "health" : "green",
    "status" : "open",
    "index" : ""test-index2,
    "uuid" : "VVU90yukSCqxs9N8VLWT0w",
    "pri" : "1",
    "rep" : "2",
    "docs.count" : "0",
    "docs.deleted" : "0",
    "store.size" : "624b",
    "pri.store.size" : "208b"
  }]
}

2. The number of indices in a page will always be less than or equal to the user provided page size.

3. The pagination would be agnostic to index creation or deletion operations which might get performed while the paginated queries are being executed. Since indices will be sorted according to the timestamps, newly created indices will always come towards the rear end of the pages, and a decision can be made whether to display them or not. If, they are not to be displayed, then the query start time (when the first page was queried) will have to stored as part of the nextToken.

4. (OPEN POINT) Response for a JSON format request is easy to define with the existing behaviour. However, with plain text format response, which is actually a (n*n) table with headers and rows, nextToken doesn't fit in.

5. (OPEN POINT) Whether to expose a parameter for user to get the list of indices in the ascending or descending order of their corresponding index create timestamps.

Implementation Details:

1. Generating sorted list of indices based on creation time. To generate a sorted list, the following approaches need to be compared: a) Sorting the list obtained from clusterState metadata at the rest layer each time a paginated query is obtained. b) Making the indices() map of the Metadata object itself deterministic/sorted. The current hashmap implementation can be replaced with say LinkedHashMap and each time a new entry (index) is put/inserted into the map, a sorted map can be re-created. c) Maintaining a cache on each node which can be updated as part of appliers and listeners being run after each clusterState update. Also, a new TransportAction (with a term version check) might also be required for rest layer to fetch that cache

2. New data members namely, nextToken and pagination response element (shards, segments, or indices) need to be introduced in the Table class which will then be read while preparing the actual RestResponse in the RestTable class.

Related component

Cluster Manager

Describe alternatives you've considered

No response

Additional context

No response

[dblock]

[Catch All Triage - Attendees 1, 2, 3, 4, 5]

[backslasht]

(OPEN POINT) Response for a JSON format request is easy to define with the existing behaviour. However, with plain text format response, which is actually a (n*n) table with headers and rows, nextToken doesn't fit in.

Curious to know how are we solving it.

Whether to expose a parameter for user to get the list of indices in the ascending or descending order of their corresponding index create timestamps.

This makes sense, helps to identify the newly created indices with descending time stamp.

[shwetathareja]

asc/ desc sorting will definitely be helpful.

[shwetathareja]

Introducing nextToken in the responseBody will change how response is parsed today