Make the health node available even if all nodes are marked for shutdown

[andreidan]

The health node makes use of the persistent tasks infrastructure.

When a persistent task is assigned we filter out the nodes that are marked for shutdown: https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/persistent/PersistentTasksClusterService.java#L346

For certain, lightweight, tasks (like the health node) it'd make sense to not have this limitation. Avoiding the nodes marked for shutdown is a nice to have, but shouldn't be a deal-breaker for task assignment if all nodes are marked for shutdown. (without a health node the health API will return No disk usage data and a status of UNKNOWN)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nmesot]

@andreidan Is this issue still relevant? I would like to work on it. We are a service provider and one of our clients would be in need of such a feature. I was thinking of introducing a config option to allow lightweight tasks being assigned to nodes marked for shutdown.

[andreidan]

Hey @nmesot Thanks for using Elasticsearch. This issue is still relevant I believe - cc @dakrone to confirm

[dakrone]

Yep, still sounds relevant to me.

[rjernst]

@andreidan @dakrone If all nodes are marked for shutdown, what behavior do you expect? Once nodes begin shutting down, they stop accepting new http requests. So whether or not a persistent task is running for health, wouldn't health api requests be rejected anyways?