confluentinc / ducktape

System integration and performance tests
16 stars 98 forks source link

Print failed test specs #301

Closed stan-is-hate closed 2 years ago

stan-is-hate commented 2 years ago

This PR adds a reporter that prints failed test symbols (including any parametrize/matrix specs) to stdout, as well as generating a test suite yaml file.

Sample output:

====================================================================================================================================================================================================================================================================================================================================================================
FAILED TEST SUITE
Test suite to rerun failed tests: /Users/stanislav.vodetskyi/github/ducktape/results/2022-04-22--027/rerun-failed.yml
====================================================================================================================================================================================================================================================================================================================================================================
FAILED TEST SYMBOLS
Pass the test symbols below to your ducktape run
'/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.failing_test' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-second","int_param":-30}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-second","int_param":10}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-second","int_param":20}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-third","int_param":-30}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-third","int_param":10}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-third","int_param":20}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"success-first","int_param":-30}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.parametrized_test@{"int_param":[10,-10]}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.parametrized_test@{"string_param":["success-first","fail-second"]}'

You can then either do ducktape results/latest/rerun-failed.yml or copy the output from stdout and paste it directly as a ducktape argument:

ducktape '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.failing_test' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-second","int_param":-30}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-second","int_param":10}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-second","int_param":20}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-third","int_param":-30}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-third","int_param":10}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"fail-third","int_param":20}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.matrix_test@{"string_param":"success-first","int_param":-30}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.parametrized_test@{"int_param":[10,-10]}' '/Users/stanislav.vodetskyi/github/ducktape/systests/cluster/test_remote_account.py::FailingTest.parametrized_test@{"string_param":["success-first","fail-second"]}'

While not as powerful as a deflake feature by @imcdo , this one allows you to edit which ones you want to rerun prior to rerunning. Yet another option for debugging flakiness :)

CLAassistant commented 2 years ago

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 4 committers have signed the CLA.

:white_check_mark: imcdo
:white_check_mark: stan-confluent
:x: nikesh0822
:x: vp-elitnet
You have signed the CLA already but the status is still pending? Let us recheck it.