Reduce usage of `TransportMasterNodeReadAction`

[DaveCTurner]

TransportMasterNodeReadAction is intended for cases where we need to collect some state held only on the elected master, for instance related to shard allocation or data stream errors. However, many TransportMasterNodeReadAction implementations work as a pure function of the cluster state, which is held on every node, so there is no need to route these requests to the master for processing. Moreover, some of these requests may be quite expensive to process in large clusters, so routing them all to the master represents a scalability bottleneck. We should reconsider each usage of TransportMasterNodeReadAction and decide whether it needs to run on the master or not. If not, we should convert them to regular local-only transport actions (e.g. using TransportLocalClusterStateAction).

Additionally, many of these actions are not currently cancellable, but they (or at least the expensive ones) should be. Experience shows that we're not great at spotting the expensive ones ahead of time, so IMO we should err on the side of caution and make each one cancellable unless we have a good reason for not doing so.

Note that attempting to route these requests to the current master does not give them any stronger consistency guarantees, because the node that does the work does not validate that it is the master before responding. It's possible that a new master has been elected, and the cluster state updated, without the responding node knowing about it.

• [ ] GetDataStreamsTransportAction (org.elasticsearch.datastreams.action)
• [ ] GetPipelineTransportAction (org.elasticsearch.action.ingest)
• [ ] TransportClusterGetSettingsAction (org.elasticsearch.action.admin.cluster.settings)
• [ ] TransportClusterHealthAction (org.elasticsearch.action.admin.cluster.health)
• [ ] TransportClusterInfoAction (org.elasticsearch.action.support.master.info)
• [ ] TransportClusterSearchShardsAction (org.elasticsearch.action.admin.cluster.shards)
• [ ] TransportClusterStateAction (org.elasticsearch.action.admin.cluster.state)
• [ ] TransportDeprecationInfoAction (org.elasticsearch.xpack.deprecation)
• [ ] TransportExplainDataStreamLifecycleAction (org.elasticsearch.datastreams.lifecycle.action)
• [ ] TransportExplainLifecycleAction (org.elasticsearch.xpack.ilm.action)
• [ ] TransportFollowInfoAction (org.elasticsearch.xpack.ccr.action)
• [x] TransportGetAliasesAction (org.elasticsearch.action.admin.indices.alias.get) https://github.com/elastic/elasticsearch/pull/101815
• [ ] TransportGetAnalyticsCollectionAction (org.elasticsearch.xpack.application.analytics.action)
• [ ] TransportGetAutoFollowPatternAction (org.elasticsearch.xpack.ccr.action)
• [ ] TransportGetBasicStatusAction (org.elasticsearch.license)
• [ ] TransportGetComponentTemplateAction (org.elasticsearch.action.admin.indices.template.get)
• [ ] TransportGetComposableIndexTemplateAction (org.elasticsearch.action.admin.indices.template.get)
• [ ] TransportGetDataStreamLifecycleAction (org.elasticsearch.datastreams.lifecycle.action)
• [ ] TransportGetDatafeedsAction (org.elasticsearch.xpack.ml.action)
• [ ] TransportGetDesiredBalanceAction (org.elasticsearch.action.admin.cluster.allocation), see https://github.com/elastic/elasticsearch/issues/101805#issuecomment-1822232998
• [ ] TransportGetDesiredNodesAction (org.elasticsearch.action.admin.cluster.desirednodes)
• [ ] TransportGetEnrichPolicyAction (org.elasticsearch.xpack.enrich.action)
• [ ] TransportGetIndexAction (org.elasticsearch.action.admin.indices.get)
• [ ] TransportGetIndexTemplatesAction (org.elasticsearch.action.admin.indices.template.get)
• [ ] TransportGetJobModelSnapshotsUpgradeStatsAction (org.elasticsearch.xpack.ml.action)
• [ ] TransportGetJobsAction (org.elasticsearch.xpack.ml.action)
• [ ] TransportGetLicenseAction (org.elasticsearch.license)
• [ ] TransportGetMappingsAction (org.elasticsearch.action.admin.indices.mapping.get)
• [ ] TransportGetRepositoriesAction (org.elasticsearch.action.admin.cluster.repositories.get)
• [ ] TransportGetSettingsAction (org.elasticsearch.action.admin.indices.settings.get)
• [ ] TransportGetStoredScriptAction (org.elasticsearch.action.admin.cluster.storedscripts)
• [ ] TransportGetTrialStatusAction (org.elasticsearch.license)
• [ ] TransportIndicesShardStoresAction (org.elasticsearch.action.admin.indices.shards)
• [ ] TransportPendingClusterTasksAction (org.elasticsearch.action.admin.cluster.tasks)
• [ ] TransportPrevalidateNodeRemovalAction (org.elasticsearch.action.admin.cluster.node.shutdown)
• [ ] TransportSimulateIndexTemplateAction (org.elasticsearch.action.admin.indices.template.post)
• [ ] TransportSimulateTemplateAction (org.elasticsearch.action.admin.indices.template.post)
• [ ] any other TransportMasterNodeReadAction implementations added since this list was created

Relates #77466

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[idegtiarenko]

I believe TransportGetDesiredBalanceAction is required to run on elected as it response contains DesiredBalanceStats that are computed in the allocator during execution. I think the same is true regarding ClusterInfo. Alternatively we can update the action to run anywhere in the cluster and read only the stats and clusterInfo from elected master.

[DaveCTurner]

I believe TransportGetDesiredBalanceAction is required to run on elected

Me too - please feel free to cross it off the list in the OP with a comment to that effect.