Ensure health/liveness/readiness probes work correctly for background task nodes

Feature Description

Kubernetes uses liveness, readiness, and startup probes to determine when an application is running, able to receive traffic, and when it needs to be terminated. We should ensure these probes work correctly in the context of Kibana nodes that are only running background tasks.

Business Value

Facilitates zero downtime rolling upgrades and rollbacks, allowing us to roll-out new features to our users more quickly while they continue to use the system without disruption. Additionally, it enables autoscaling Kibana's background task nodes based on their utilization. Customers will no longer need to manually size their Kibana nodes (less effort), resources will be used more efficiently (decreased COGS) and tasks will run with less of a delay (decreased MTTD, MTTR, etc).

Definition of Done

Kibana background task nodes are marked as running as soon as possible
Kibana background task nodes properly indicate when they are deadlocked and should be terminated

Reference Links

https://github.com/elastic/kibana/issues/158910

elastic / kibana