Add debugging for hung tests

confluentinc / ducktape

System integration and performance tests

11 stars 93 forks source link

Add debugging for hung tests #359

Closed ajfabbri closed 10 months ago

ajfabbri commented 2 years ago

To allow us to debug hard-to-reproduce "hung test" failures, this PR adds special handling of test timeouts which, for any still-running child processes:

Sends a SIGUSR1 which causes them to print stack traces of all threads.
Sends a SIGINT and wait for children processes to exit.

The goals are to provide diagnostics on where child test clients are stuck, and to allow the ducktape processes to exit, which should allow gathering logs.

CLAassistant commented 2 years ago

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

ajfabbri commented 2 years ago

Thanks for the reviews! I will push an update when I get a chance, which will:

Move hung test handling to runner.
Explain or remove ZMQ / TCP constant changes.
Add a test.
Fix neglected merge markers.
Address other comments (typing syntax, remove dead code comment, etc.)

I will go ahead and mark this as a draft until it is ready for the next round of reviews.

imcdo commented 2 years ago

Again thanks so much @ajfabbri this is really a much appreciated feature

imcdo commented 1 year ago

@ajfabbri any update on this?

ajfabbri commented 10 months ago

Hi there. I am no longer working on Kafka stuff so I'm unable to work on this. I will close this PR but I welcome anyone to take over and finish developing the patch.