search

r-lib / testthat

An R 📦 to make testing 😀
https://testthat.r-lib.org
Other
888 stars 318 forks source link

FR: Provide a way to run unit tests in random order #1942

Open IndrajeetPatil opened 6 months ago

IndrajeetPatil commented 6 months ago

Preamble

Running unit tests in random order can help uncover hidden dependencies and assumptions within the tests themselves. When tests are run in a predetermined order, it's possible for dependencies between tests or shared state to go unnoticed. Randomizing the order can expose these issues, ensuring that each test is truly independent and capable of standing on its own.

In Python pytest-random-order plugin for pytest provides this functionality.

Feature request

Currently, {testthat} always runs tests in the alphabetical order (except when Config/testthat/start-first is set), and it would be great if it can also provide a way to run the tests in randomized order to detect coupling between different tests.

Not sure about the possible API: It could either be a new function or a parameter to test_dir(), which is then exposed via devtools::test().

hadley commented 5 months ago

Hmmm, interesting idea, and it would be easy enough to implement by calling sample() on the parsed vector.

To me feels like it would be best exposed as an argument to devtools::test().

...

Oh are you thinking random file order, or random order within file, or both?