Submissions WG presentation (2023-01-13)

[dgkf]

Hi all,

Today we presented at the R Consortium's Submissions WG meeting. What is particularly impressive about this WG is that they have pretty extensive participation from health authorities and have a pipeline in place to perform pilots where they ship deliverables to health authority reps who document their findings when running through the pilot. Generally these are technical tasks like delivering an in-house R package through the FDA's ESUB portal or spinning up a shiny app on the FDA's scientific machines used for review.

We presented on our mission, uncertainties and where we could use participation and feedback. Interestingly, we got a few helpful tidbits from the health authority participants.

• There seem to be a consensus opinion at the FDA that the R Validation Hub's recommended practices are a good baseline to follow.
• Concerns around use of R packages are generally driven by IT from a security perspective, with a past example being a buffer overflow issue in the haven package.
• There was later mention about statistical accuracy, but it seemed to be understood that there's always acknowledgement that software is used "as is"

Now this brings me to some of my personal take-aways

• I still think we can move forward with some questionnaire/wireframe pilots, but this discussion will probably color what questions we're most interested in asking
• This might suggest that we should focus less on exact criteria, and more on how we provide a forum for discussions about things like security vulnerabilities or statistical accuracy. (the nodejs ecosystem jumps to mind, which has a reporting mechanism for known vulnerabilities built into their tooling for fetching packages).
• Since these are more fundamental challenges regarding transparency, we should grow closer ties to the R Consortium's Regulatory WG

Overall I would say this was an enlightening discussion that will help set us off on the right path

---

tagging @kkmann and @borgmaan - anything else I might have missed?

[borgmaan]

@dgkf - I agree the discussion was very enlightening and big thanks to you to leading the presentation and discussion to represent the work stream. I think it went great.

I think you summarized the key points quite well. I do think I disagree one of your takeaways:

This might suggest that we should focus less on exact criteria, and more on how we provide a forum for discussions about things like security vulnerabilities or statistical accuracy.

I thought the discussion cemented the need for a cross industry-repository of QC criteria, and, additionally, that riskmetric/R Validation Hub criteria were a great set of baseline metrics to start with given:

There seem to be a consensus opinion at the FDA that the R Validation Hub's recommended practices are a good baseline to follow.

Keeping our vision/goal in mind of easing the burden of R adoption -- especially for organizations that may not have extensive resources to develop in-house validation solutions -- I'm wondering if building something like a central store of riskmetric runs is a good technical fit for the pilot (think riskmetric as a service (RMaaS)).

I think this would be very valuable and would have a lot of synergy with the existing work being done on the riskassessment application. If we are thinking of the "PharmIKEA" approach, a reasonable path would be to instuct individuals to leverage riskassessment to begin the process of partnering with their internal IT and Quality stakeholders on approving packages that meet their org's risk profile.

A centralized source of riskmetric runs on a broad swath of packages (both current and archived) would be a significant value add for users of the riskassessment application. It would allow organizations to seed their initial databases with a broad catalogue of packages (and versions) and keep them in sync over time.

Additionally, I have seen multiple issues on both riskmetric and riskassessment highlighting a desire for longitudinal assessments of risk scores on packages. A centralized store of these metrics would ease the burden of implementing some of these features.

Centralizing the metric store also allows organizations to build their own riskassessment-style layer for decision making if they prefer to do so.

Related... I think the security component was a good call out. Perhaps, if it does not currently exist, a security scan is something that would be worth adding to riskmetric to feed into this theoretical ecosystem.

I know this is a bit out of left field, but wanted to get some of these thoughts out there as they have been churning in my head after being exposed to some of these different initiatives.

[dgkf]

You raised a lot of good points, but just to reply quickly to one of them there is the oysteR package for security vulnerability scanning. That might be a good starting point for surfacing some measure of security assessment.