Detect deadlocks when running hot_threads

[ywelsch]

A node that has deadlocked threads can exhibit all kinds of issues, which can be difficult to explain if not looking for this exact cause (see #41418). It would be convenient if hot_threads, which already provides a lot of useful information about the threads, would also provide information on whether there's a deadlock on the respective node.

Funnily enough, our code base already has a DeadlockAnalyzer (introduced by Shay a very long time ago, but deactivated very early on), so this would make use of this class again, which has been laying dormant for 8 years.

[elasticmachine]

Pinging @elastic/es-core-infra

[jasontedor]

We discussed this during our weekly fix-it meeting. We think this is a worthwhile addition although we are doubtful about overloading hot_threads for this purpose. Instead, we discussed the possibility of adding a dedicated threads API that would give basic reporting about threads (stacktraces, etc.) and could include deadlock detection, basically bring jstack into Elasticsearch.

[williamrandolph]

I would like to raise this ticket for discussion with the Core/Infra team to decide if a dedicated threads API is "high hanging fruit" or if it is something that should be on our roadmap.

[williamrandolph]

We discussed this today in core/infra. We don't think this will get on the roadmap in the short term, but we still see the benefit in doing it, so I've put the "high hanging fruit" tag on it.