[BUG] Option in NodesStatsRequest to return per-shard stats in response

[rajiv-kv]

Describe the bug

NodesStats contains per-shard stats information from the node (NodeStats#indices). Admin API /_cat/nodes make broadcast calls to all nodes in cluster using TransportNodesStatsAction to retrieve stats information from nodes. However the per-shard Stats information returned by Transport action is never consumed .

In large clusters, the per-shards information lead to memory allocations there-by triggering GC activity which consumes CPU cycles.

Memory Allocation Profile for /_cat/nodes API where the allocations are for IndexShardStats

This also causes increase in the latency of API as the per-shard stats is accumulated on the co-ordinator node after retrieving it from all the nodes in cluster.

Related component

Cluster Manager

To Reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Introduce option in NodeStatsRequest which specifies if per-shard stats information needs to be returned in response by TransportNodesStatsAction. We could unset the attribute in usecases which do not require per-shard stats there-by avoiding memory allocations.

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[rwali-aws]

[Triage - attendees 1 2 3 4 5 6]