New Plugin Extension to Find Flaky Tests (Enhancement PR)

yashsaboo commented 3 years ago

Order-Dependent(OD) tests are flaky tests whose results can differ depending on the order in which the tests run. An order-dependent test consistently passes when run in one order but then consistently fails when run in a different order. [Definition and plugin extension inspiration from Research Paper]

We have come up with a new extensions to this plugin that helps in identifying the OD Flaky test which features two flags: --flaky-test-finder= where repetition_number > 1 --flaky-test-log-path=

--flaky-test-finder When this flag is enabled, it identifies the flaky test that satisfies the above definition.
--flaky-test-log-path This flag specifies where the flaky test logs are flushed.

Contributors: Yash Saboo Nirupam K N

jbasko commented 3 years ago

Thank you for the submission. I will have to spend some time on this to consider the merit of including this in the project. I might not be able to find the time before Christmas. Feel free to remind here if there is no update by then.

Nirupamkn commented 3 years ago

That sounds great! Merry Christmas :) @jbasko

jbasko commented 7 months ago

Apologies for taking 3 years to look again into this. I think your contribution does not belong to this simple plugin.

pytest-dev / pytest-random-order

New Plugin Extension to Find Flaky Tests (Enhancement PR) #43