Feature Proposal: Additional test roxygen headers for more granular metadata

[dgkf-roche]

In my experience within the regulated industry space, many software teams are looking for more granular documentation about tests. Broadly, this could also provide a good avenue for adding other test qualities.

At my place of work we've explored some early POCs for adding roxygen2 headers to testthat::test_that blocks and wanted to float the idea upstream.

#' @meta author dgkf
#' @meta feature cli
test_that("cli printing works as expected") {
  ...
})

This is largely inspired by a pattern for using pytest marks for attaching test metadata:

@pytest.mark.metadata(
    author="dgkf",
    feature="cli"
)
def test_cli_printing():
    ...

Although metadata is certainly the focus of this proposal, adopting roxygen headers for tests could have other benefits, such as allowing for better-documented tests with separation of succinct title and longer-form description for complicated test scenarios and possibly using it as a more consistent location to tag tests as "skip on" or other flags.

For the sake of discussion, a reckless re-imagining of the test api could look something like:

#' cli printing works as expected
#'
#' Test that a sensible cli is displayed regardless of terminal capabilities, 
#' testing fully ansi-capable ttys, dynamic last-line capable tty renderers
#' like RStudio and a simplified ascii-only output.
#'
#' @meta author dgkf
#' @meta feature cli
#'
#' @skip-on CRAN, CI
test({
  ...
})

[hadley]

I think this is an interesting idea, but I think unfortunately it's out of scope for testthat.

[dgkf-roche]

Thanks for confirming! This was my suspicion, but I wanted to float it here first as to not split the ecosystem unnecessarily.

We've got a little POC that implements a new reporter for this - effectively a JunitReporter that parses the roxygen headers and inserts them into the junit output. I'll see if we can get it open-sourced now knowing that this is something you feel is best served by a supporting package.