[Planning] Gather updates from each workstream for Posit Conf '24

[aclark02-arcus]

Greetings all,

In preparation for the R/Pharma in-person Summit hosted @ Posit Conf this year, we plan to present a brief summary of progress made in the last 6-months for each R Validation Hub workstream. Assume each workstream will get a slide. As such, please comment below with 2-ish bullet points that provide a high level summary of your recent work. If you have an (hi-res) image you'd like to share, please post it in this thread also. I'll make sure it makes it to the slide deck. Note: the quarto slides aren't yet available, but I'll start preparing those soon. Note # 2: the R/Pharma conference committee wants us to give a large quantity of time to the Regulatory R Repo Workstream, so no need to give a summary here.

Requested by:

• Optimistically, by our next exec meeting, July 9th
• At the latest, July 15

Tagging all other workstream leads:

• {riskmetric} @emilliman5
• {riskassessment} @jthompson-arcus @aclark02-arcus
• Comms & Case Studies @pharmaR/ws-communications

[aclark02-arcus]

Met with Comms team 7/11, @pharmaR/ws-communications want to highlight two things:

• New website
  • Big refresh - easy to navigate
• Generate interest in community meetings / case studies
  • side benefit: increase interest in R Consortium / R Val Hub
  • upcoming community meeting dates:
    • August 20
      • Speaker: Brid Roberts, Principal Statistical Programmer, Novartis
      • Title: Analyzing change in assessed risk across package releases’
      • Abstract:

        The Software Open Source (SOS) team manages and executes the risk assessment process for R package validation at Novartis. The team uses an internally developed R package to classify the risk of each package as "low", "medium", or "high".

        We analysed the risk assessment data over two time points to determine the impact on the assigned risk categorisation for packages with AND without version changes.

        In this talk, we showcase the risk assessments over time, the causes of any risk class changes, and their impact on various teams within our organizations as a result.

[aclark02-arcus]

Met with @emilliman5 on 7/12 to discuss. Here are some meeting minutes:

Not required, but consider providing update on last year's areas dev focus:

• Ease of use
  • is near completion
• Metric completeness
  • made some progress
• Modular additions
  • back burner
• Focus on metrics & scoring
  • working on metrics now
  • More generic metric functions for robustness & transparency
  • Making scoring for custom weighting - just need to merge it
  • BUT, big news: looking to form guidance on scoring

Risk-tools Discussion topics

• Should {riskmetric} focus on just the metric assessments?
  • Ideally, {riskmetric} should solely generate evidence
  • Repos group (or some other WG or even the decision automation module in the {riskassessment} app) can help end users draw a line in the sand on scoring
    • That is, {riskmetric} should be used as building block for these other entities
  • Reminder: you technically don't need a "risk score" to qualify pkgs for your environment
    • Write tests to make sure it does what you expect
    • Use the assessments as a grading rubric
      • Most companies do this according to our case studies
        • include stats from 2022 recordings
      • Example: a package is qualified for GxP use if test coverage > 75% & has vignettes, etc. Not if it has a risk score < 0.3
  • Don't worry, but development team will never fully remove (deprecate) the scoring functions. Still want to showcase "how users could" execute the process end-to-end.
  • At the end of the day, scoring is subjective - should be decided as R Validation Hub as a whole.
    • Solicite participation in a new strategy group to provide guidance
    • Forum to discuss the philosophy of decision making on risk criterion

[aclark02-arcus]

FYI - workstream updates can be (& will be) added here: https://github.com/pharmaR/events-positconf2024/blob/main/_r_val_hub_workstream_updates.qmd

Will close this issue once all their content has been updated in the slides

[aclark02-arcus]

Key Visuals

• Use @dgkf Qmd slides. Doug, I took a quick snapshot of this when you presented it, but would love if you could share the code with me that generated them so I can integrate into our slide deck?

[aclark02-arcus]

Just separating this out from the comment above: we have the option to do an in-person survey question through menti, or pre-survey attendees with these questions:

• All sponsor companies need to perform a risk assessment of their software tools. We'd like to better understand what we have to gain by moving this work into the public sphere. How many full-time employee equivalent (FTE) are committed to risk assessment within your organization?
  • ideally numeric response, at 0.25 or less increments; if multiple choice, perhaps >10; >5; >3; >1; 1 or less
• Does your organization use {riskmetric} for risk decision-making? If so, how?
  • No
  • Yes, we use the default summarized scores
  • Yes, we use the default metric scores, but apply custom logic
  • Yes, we use the assessment information to define a decision tree
  • Yes, we use the assessment information and make a decision on a case-by-case basis
  • Other (free text)

[aclark02-arcus]

Posting for @Crosita's sake, in regards to drafting the Sept Newsletter for the R Consortium.

{riskassessment} app update

• New features:
  • Automate "risk decisions" based on {riskmetric} quality assessment values
    • Admin users can define quality rules in the app, and any time a user upload a package to the database, a decision will be made without the need for manual review.
      • Example: If a package has more than 200k downloads annually, automatically categorize as "low risk".
  • New module: the "Function Explorer"
    • Explore any function exported from a package. In one easy to use interface, view it's
      • Source code
      • Rendered help docs
      • Unit tests
    • Special thanks to GSK contributors for donating this code!
• Roadmap: going scoreless
  • Most organizations don't really make actionable decisions from {riskmetric}'s overall package score. In fact, it can sometimes deter our attention from the more meaningful quality metrics or even lead to unnecessary confusion or bias. As such, we want Admin users to make the decision whether they want this quantitative metric to be displayed in the app at all.
  • Consider joining development
• New Deployment:
  • app.pharmar.org/riskassessment/
  • Why? A community experiment for any org to collaboratively provide feedback on a specific R pacakge
  • Help us make community-driven decisions on each pkg (Low, Medium, or High risk)

R Validation Hub "Strategic" plans

• Collecting GxP package lists
  • what have you qualified for late stage analysis?
  • See #52