Closed dgkf closed 6 months ago
@dgkf - I agree the discussion was very enlightening and big thanks to you to leading the presentation and discussion to represent the work stream. I think it went great.
I think you summarized the key points quite well. I do think I disagree one of your takeaways:
This might suggest that we should focus less on exact criteria, and more on how we provide a forum for discussions about things like security vulnerabilities or statistical accuracy.
I thought the discussion cemented the need for a cross industry-repository of QC criteria, and, additionally, that riskmetric
/R Validation Hub criteria were a great set of baseline metrics to start with given:
There seem to be a consensus opinion at the FDA that the R Validation Hub's recommended practices are a good baseline to follow.
Keeping our vision/goal in mind of easing the burden of R adoption -- especially for organizations that may not have extensive resources to develop in-house validation solutions -- I'm wondering if building something like a central store of riskmetric
runs is a good technical fit for the pilot (think riskmetric
as a service (RMaaS)).
I think this would be very valuable and would have a lot of synergy with the existing work being done on the riskassessment
application. If we are thinking of the "PharmIKEA" approach, a reasonable path would be to instuct individuals to leverage riskassessment
to begin the process of partnering with their internal IT and Quality stakeholders on approving packages that meet their org's risk profile.
A centralized source of riskmetric
runs on a broad swath of packages (both current and archived) would be a significant value add for users of the riskassessment
application. It would allow organizations to seed their initial databases with a broad catalogue of packages (and versions) and keep them in sync over time.
Additionally, I have seen multiple issues on both riskmetric
and riskassessment
highlighting a desire for longitudinal assessments of risk scores on packages. A centralized store of these metrics would ease the burden of implementing some of these features.
Centralizing the metric store also allows organizations to build their own riskassessment
-style layer for decision making if they prefer to do so.
Related... I think the security component was a good call out. Perhaps, if it does not currently exist, a security scan is something that would be worth adding to riskmetric
to feed into this theoretical ecosystem.
I know this is a bit out of left field, but wanted to get some of these thoughts out there as they have been churning in my head after being exposed to some of these different initiatives.
Hi all,
Today we presented at the R Consortium's Submissions WG meeting. What is particularly impressive about this WG is that they have pretty extensive participation from health authorities and have a pipeline in place to perform pilots where they ship deliverables to health authority reps who document their findings when running through the pilot. Generally these are technical tasks like delivering an in-house R package through the FDA's ESUB portal or spinning up a shiny app on the FDA's scientific machines used for review.
We presented on our mission, uncertainties and where we could use participation and feedback. Interestingly, we got a few helpful tidbits from the health authority participants.
Now this brings me to some of my personal take-aways
nodejs
ecosystem jumps to mind, which has a reporting mechanism for known vulnerabilities built into their tooling for fetching packages).Overall I would say this was an enlightening discussion that will help set us off on the right path
tagging @kkmann and @borgmaan - anything else I might have missed?