Documentation requirement

[mmengelbier]

A general question regarding what compliance documentation is expected as a product of producing package quality indicators, in the form of factors and metrics. Industry best practice would expect that there is a predefined process, i.e. pipleine description, and documented evidence on the result from producing said quality indicators using that pipeline.

Using the case of organization/company A that intends to use the "database" of wg quality indicators in their internal package risk assessment and validation, would they expect documentation that

• demonstrates/describes the provenance of those indicators, i.e. package, version, methods, inputs, execution environment, outputs, etc.
• documents the derivation of indicators, e.g. execution logs
• provides a basis for reproducing estimates
• provides traceability in the chain of actions and events resulting in those indicator estimates being published
• qualifies the environment as adequate enough or fit-for-purpose to be used in producing indicator estimates

As a summary, what documentation would be need to establish trust in the indicators maintained in wg repository ? Would that be the same level of documentation expected to support company A's risk-based validation and compliance argument? Would company A expect that the indicators are estimated using a qualified pre-specified controlled environment?

[Zhenglei-BCS]

Documentation:

1. list of packages in the bundle together with version.
2. Security considerations if any.
3. Installation/deployment plan
   • Detailed installation steps to reproduce the wg repository or a reference image.
   • Installation problems and how to handle them.
4. System tests: procedures and report (not sure if this applies to the wg repository, as this is mainly for risk characterization, not the actual testing strategy):
   • Risk assessment. (riskmetrics, which metrics are used.)
   • what kind of validation tests has been designed and the test output.
   • excluded unit tests and unit test output.
   • When the tests fail, how to update the tests or the system.
5. System requirements: additional software like Git, minimum specification(CPU, RAM, Storage) for the server or instance.
6. Updates and changes from previous version.

[Zhenglei-BCS]

Validation reporting for a single package:

• just document the facts, decision of pass or fail should made afterwards, maybe in the summary table

Action Points: Start a skeleton for a template rmd for https://github.com/pharmar/pharmapkgs @Zhenglei-BCS

Table of Conents

• [ ] Risk metrics report (Package information summary)

• [ ] Risk score / characterization

• [ ] Unit Test Report

• [ ] System Test Protocol

• [ ] List of Test Cases (Definition & Measurement of success)

• [ ] Test Cases Execution Report

  • [ ] Execution: Environment

    Styles

• [ ] R Validation Hub logo

• [ ] R-repo wg word document?

• [ ] Do we generate multiple documents (Risk Assessment, Test Approach (2 levels), unit test report, system test protocol, list of test cases and their definitions, test cases execution report) or combined into one document?

• [ ] Other considerations?

[Zhenglei-BCS]

Example of skipped test for testing riskmetric package:

devtools::test() ℹ Testing riskmetric ✔ | F W S OK | Context ✔ | 2 | assess_dependencies
✔ | 7 | assess_export_help
✔ | 2 | assess_has_bug_reports_url
✔ | 5 | assess_has_examples
✔ | 2 | assess_has_news
✔ | 2 | assess_last_30_bugs_status
✔ | 2 | assess_news_current
✔ | 14 | assess
✔ | 1 | metric_score_labels
✔ | 1 | metric_score_range
✔ | 31 | pkg_ref
✔ | 3 0 | snapshots

══ Results ═══════════════════════════════════════════════════════════════════════════════════════════════════════════ Duration: 3.2 s

── Skipped tests (3) ───────────────────────────────────────────────────────────────────────────────────────────────── • snapshots appear to be failing due to random order of pkg_ref fields (3): test_snapshots.R:7:5, test_snapshots.R:20:5, test_snapshots.R:35:5

[ FAIL 0 | WARN 0 | SKIP 3 | PASS 69 ]

[Zhenglei-BCS]

About the testing framework, Magnus mentioned that there should be multiple reference images for different industry, GCP / GXP. And it should be agreed that as long as the reference image has passed all the testing and documented all test results, each organization could safely pull the image into their own environment.