Validation Report generation approach

[thebioengineer]

I wanted to document how we wanted to approach building up the validation report since it is comprised of multiple pieces that depend on the users goals whether they are

• validating an internal package from within it
• validating an internal package from outside it
• validating an external package

We could take the "don't edit this" approach of roxygen2 x namespace relationship, Where the users document the order in which they expect things to appear in the report like the bookdown yaml approach.

Or the copy into clipboard and its up to the user to insert a la badges in readme from usethis.

or automatically add it to the end of the file,

or fill in the document with everything we can think of and the user needs to drop/remove unexpected content.

I can see value in all these approaches, so I wanted to air it out

[mariev]

hmmm - that's a good point, What would be available only in the source package form? Is it only the function author details that can't be copied as part of /validation/?

Thinking about /validation/ as the set of files that get preserved regardless of method means the report will always have access to the requirements/test cases/test code. So the report RMD can always scrape those roxygen blocks during render. The test code will be evaluated when the report is rendered, so that code won't vary between scenarios - same for any code retrieving validation environment details. Signature block info is captured via .validation config, so that file should be considered part of the set of files that are ported around.

[thebioengineer]

I have been thinking about that as well. part of the build process when the validation is happening from within the R package will need to copy the .validation into inst/ and will also scrape the roxygen blocks and store them inside inst/validation/R or something

[elimillera]

I really like the YAML approach. We could almost consider this like building a pkgdown page. In that, you can specify different styling options and exactly specify what you want to appear, in what order, and how it should be grouped

[thebioengineer]

to move some comments from #65:

What will rendering look like?

• We are using validation.yml to determine the contents and order of elements in the validation report. This idea is similar to the bookdown.yml approach. in bookdown, there are two schools of thought, merge-knit and knit-merge. For ease, it is likely the best method to go with the merge then knit method. Meaning contents will be scraped and combined together into a single document, then actually rendered into the report.

How to do this?

• My first thought is that we will need to have a shell validation report live in the vignettes folder, that will really function to call and report the contents. This way we preserve the vignette building on install. Functionally, it will be "empty", and likely look something like:

---
title: Validation Report
author: Validation Lead
date: "`r Sys.Date()`"
output: pdf_document
---

```{r include = FALSE}
vt_generate_validation_report()

  - `vt_generate_validation_report()` (STILL OPEN TO NEW NAME) would be the workhorse to read the file order from validation.yml, and start printing the contents "asis". The child documents (other than req, test_cases) are to live, and we concluded that our approach could be that the contents could live anywhere within the `{{working directory}}/validation` directory and the function would search out the file.

Questions I have:
 - How to approach the test code, since it is written as an R script, we would need another file to provide the rendering/formatting, correct?
 - will we have a new suite of "vt_use_*" for the report infra: `vt_use_sig_table`, `vt_use_env_report()`, etc?

[mariev]

My understanding is that, regardless of validation paradigm, a validation report ultimately is a single .RMD document. {valtools} users can create this by:

1. Writing the .Rmd validation report directly, calling helper functions like vt_run_test_code_file in specific R chunks and adding whatever custom combo of R chunks/markdown/LaTeX/HTML code they are comfortable using. This is especially useful for developers who may need customize report formatting for their org's needs e.g. if SOP dictates that signatures are captured using a separate form.

2. Using a {valtools} function to create the .Rmd validation report using validation.yml details i.e building up from the files listed in validation_files list.

A couple notes on this process:

• For re-validation i.e. anything downstream from initial release, we need a system that has the flexibility to rebuild from components and retains the full .Rmd to play nicely with build_vignettes=TRUE, so that validate on install is possible.
• Writing the .Rmd source can be a separate activity from validation. This is especially true for the "validate after install" or "validate after environment change" crowd. There is a certain additional level of trust that comes from knowing that the report being rendered to pdf/html is the same single validation source file vs. a collection of files.

[thebioengineer]

oohh...so vt_generate_report would populate based on the validation.yml an rmd that would then serve as the report?

[mariev]

I think if our goal is to facilitate "validate on install" as the default paradigm, we need to focus on how {valtools} functions make writing that vignette content pain-free.