testthat chunks for R Markdown documents

[jcheng5]

Background

I've been thinking about a dedicated R Markdown chunk type for tests (e.g. testthat). These can be sanity checks of any kind, but particularly to detect problems with your data.

I have some R Markdown reports that run daily; it pulls down logs from S3 and does some basic aggregation, and draws some plots. Every once in a while there is a failure in the ETL process that generates the logs, so a day of data goes missing; or some new factor level is introduced that I don't expect. Often these problems don't cause the report to break in an obvious way (especially if the report only shows aggregated/summarized data).

Proposal

I would like to have a ``{tests} chunk type (or a mode on the {r} chunk type) that lets me write whatever tests I want. Check for missing dates, values that are unrealistically out of the norm, unexpected categorical values, CSVs having the wrong number of columns, unexpectedNA`.

If the tests warn or error, then you a very obvious warning/error indicator is emitted in the document, with UI you can expand to reveal more details, including the test code.

If the tests all pass, then the chunk does not show up (by default--maybe a global checkbox at the top or bottom to reveal all test chunks) (or maybe it can optionally show a "51 of 51 tests passed" with a green check mark or whatever).

---

If you write R Markdown reports regularly, does this sound like something you'd actually find yourself using?

[jcheng5]

Oh, just seeing assertr now.

[cboettig]

Great idea. I'd be curious to hear more discussion about this kind of workflow around RMarkdown reports in general (e.g. do you automate this generation with a CI integration or some other approach? is this worth doing on notebooks that aren't being run on dynamic/changing data? Do you use/need some kind of notification system to alert you if a report fails to build, e.g. like CI warnings / errors?).

Haven't tested it, but can't we get rather close to this behavior with existing knitr options though? e.g. if we write a chunk such that it has no output if the tests pass, but throws warnings or errors when they don't (e.g. the standard testthat behavior) we can set error = TRUE on such a chunk so that it does not stop execution, and echo = FALSE if we don't want to display this code. Maybe there's some behaviour I'm not thinking about properly, certainty could be more elegant.

[jcheng5]

Haven't tested it, but can't we get rather close to this behavior with existing knitr options though?

I think you can get close enough that it's useful to write tests today. But treating tests as first-class citizens would give us the ability to improve both the authoring syntax, and the UI on the output side.

[karthik]

I love the idea @jcheng5! In addition to assertr, there are also several other options (ensurer, visdat, pointblank, Comparison blog post) that could leveraged to pass before running downstream chunks.

Some UI ideas from pointblank would be worth consulting if this were to become a first class citizen on rmarkdown.

[noamross]

I really like this idea. Over in #5 I was thinking along the lines of exporting the Rmd environment and having a separate testing script, but this seems more intuitive. I would also consider having a chunk type for non-boolean tests, such as diagnostic plots. Maybe ```{check}? These could also be rolled up into a hidden but expandable section of the document.

Another thought - how might one assess test "coverage" in such a case? One possibility is counting the % of objects created in the document that are tested, or the % of objects which are printed or plotted? With something like assertr one might be able to measure the fraction of a data frame that has been tested. These are all crude measures, but so is % of lines of code, and its still quite useful.

[hadley]

It's also worth consider if testthat could provide more infrastructure for this usage. I don't think it'd ever be as good as a tailored solution like assertr, but it's worth exploring a little (especially since recent versions are very pipeable)

[rmflight]

There are 4 of us working on this on the main tables in the big windows.

@jcheng5 @batpigandme @christophertull @rmflight

[stefaniebutland]

testrmd repo: https://github.com/ropenscilabs/testrmd