R-Consortium Repos WG Discussion (2024-04-01)

[dgkf]

While we have standing members (@wiligl, @borgmaan) joining the R Consortium Repos WG, for this particular session we'll also join with Coline as workstream lead and myself as RValHub lead.

We were asked to prepare a set of regulatory requirements and goals that we have for our work to help articulate our goals. I'll try to aggregate suggestions as they're mentioned

Attendees

@wiligl, @borgmaan, @dgkf, @Crosita

Prior Documents

• Early on we provided a wishlist to the Repos working group, which still looks like it applies today.
• MVP of client-side package filters
• Current approach is looking to model r-hub/repos

Overview of Goals

Provide a CRAN-like repository, with CRAN-like checks for package quality that satisfies regulated industry needs for documented proof of the reliability of analytic software.

Requirements

We have a few fundamental requirements that are unmet by CRAN today, representing barriers to adoption in a regulatory setting.

It's important to note that our target audience is non-experts - often non-R users from an IT support function or validation role who often see opportunity for error as an insecure system. Even if challenges can be solved, unless the solution is easy, it may still be perceived as faulty.

• Availability Snapshotting, avoiding changing tarball url upon archiving. Somewhat solved by tools like renv, devtools::install_version, MRAN, PPM snapshots, etc. Ideally would have a more "official" mechanism that avoids requirement to use specific tooling or vendor services.
• Build Transparency In a regulatory setting, if packages were to be used directly, we'd need to be able to assess any risk between how the package performs on a build system and how we would expect it to perform on a sponsor system or health authority system.
• Metadata Probably the biggest requirement, we'd ideally have enough information to characterize the "risk" of a package. Notably, this might include test coverage, usage statistics, and other checks for best practices. Note that the regulatory space has different risk tolerance for different types of analysis, so there's no single cutoff that we might apply. For this reason, we've explored mechanisms of making this cutoff somewhat dynamic either as a dynamic server-generated PACKAGES index, or as a client-side available.packages filter.
• Security From feedback we've received from health authorities (specifically from their IT security folks), vulnerability scanning is a key capability slowing their adoption of R. Examples of problematic packages would have been discovered via oysteR scan.
• Availability
  • Minimize gap between private use of packages and health authority reviewer (FDA) use. Likely means that the bulk of packages would be served centrally and publicly.
  • In addition, private enterprises will certainly need to supplement this cohort with additional private packages, and we'd like for the solution to be able to be privately-hosted to supplement a public cohort.
  • Possibly need to be able to support non-CRAN public packages as well

Opportunities for Overlap

• ROpenSci R-Universe
  • Amazing resource for hosting non-CRAN packages.
  • Presents challenges for asserting qualities of explicit versions of packages as derived in explicit test environments
  • Looking for Feedback: If we wanted to supplement a universe with checks/metadata, how might we integrate those checks with a universe (proposal: an "extension" that runs additional actions in the universe repo in addition to the Build & Deploy actions?)
• r-hub/containers
  • Closing the gap with transparency of CRAN test evaluation.
  • A likely candidate for the images we'll use to provide check results.
• r-hub/repos
  • Supports repo hosting in a format that compliments existing designs for providing check metadata (via supplementary fields in PACKAGES)
  • Looking for Feedback: Ideally we'd have logs/output from checks executed in r-hub/containers provided alongside PACKAGES. Is a fully r-hub-powered CRAN-like check suite a possibility? How would the output of checks be provided?

[Crosita]

@wiligl @borgmaan please see what we have so far and contribute. Three sections:

1. Overview of Goals
2. Requirements
3. Opportunities for Overlap

The Reg R Repo workstream is asked to share requirements on the April 1st meeting with the Repositories working group.

[borgmaan]

Read through it and don't have anything to add. Looks great @dgkf!

[dgkf]

Relaying some feedback that we got from the R Consortium team giving this a pre-read. Question raised:

• How would the repository be maintained?
• Would the repository be private, company/submission-specific or public?

I can certainly address this above.

[wlandau]

From today's discussion, R-multiverse is an area of overlap: https://r-multiverse.org/. The project is very new, still in its prototype phase. Right now, it implements a community-curated R-universe of package releases, which effectively amounts to a QA repository. We are exploring solutions for production which layer on top of the QA universe.