Incongruity of test_local() argument, argument usage, and documentation

[hdhshowalter-lilly]

test_local() includes the argument path, and the corresponding argument documentation states "Path to directory containing tests." This is consistent with the documentation for the path argument of test_dir(). In practice, however, test_dir() can be executed against test files located in directories other than test/testthat, but test_local cannot; https://github.com/r-lib/testthat/blob/main/R/test-package.R#L64C1-L64C1 forces the tests to live in test/testthat. I do see that this expectation is referenced in the "special files" vignette (https://github.com/r-lib/testthat/blob/main/vignettes/special-files.Rmd#L27C1-L27C13), but then I'm not sure what purpose path serves within the context of test_local().

For strategic reasons, I have a separate set of tests that I would like to store in a different directory. I can get these to run by utilizing test_dir() (or test_file(), or the internal function test_files()), but not test_local(). Thus I would personally love path to function the same way in test_local() as it does in those lower-level functions.

Tangential nice-to-have: I would also like to be able to able to give my separate test set a (common) prefix other than "test-" or "test_", but find_test_scripts() (https://github.com/r-lib/testthat/blob/main/R/test-files.R#L375) prohibits this. It would be cool if that expectation was the default, but could be optionally skipped over in favor of using the filter argument directly (e.g., to identify a collection of files that starts with some other prefix).

[hadley]

This is a problem in the documentation for test_local(); it's not possible to change the function itself because it's used in a bunch of places (most importantly in devtools::test()).

It generally sounds like you are trying to fight against testthat's conventions; I'm sure you have your reasons but you will definitely find life easier if you stick to the defaults.

[hdhshowalter-lilly]

My hope was that test_local() could be updated so that the current behavior remains the default, thus maintaining backwards compatibility. But perhaps given the goal of running each test in a clean environment, that is easier said than done.

To provide a bit more context: the second set of tests I alluded to falls outside the scope of "unit testing", which is why I'd like to organize them in a separate directory. (My unit tests absolutely utilize testthat's conventions.) Indeed, it's quite likely I would not even include them in the build of the package! However, testthat is still a great engine for running these tests because I'm able to format the investigation into test_that() statements with various expectations; running them using an alternative method would be needlessly re-creating the wheel. It's really just the desired location of these tests that is preventing everything from being totally seamless.