Highlight differences between new & old assessments

pharmaR / riskassessment

Collaborative Deployment: https://app.pharmar.org/riskassessment/ Risk Assessment Demo App: https://rinpharma.shinyapps.io/riskassessment

https://pharmar.github.io/riskassessment/

Other

101 stars 27 forks source link

Highlight differences between new & old assessments #254

Open AARON-CLARK opened 2 years ago

AARON-CLARK commented 2 years ago

Whenever a package get's re-assessed (when weights change or when pkg version changes), it'd be useful to...

(1) compare the prior & current risk metric score, and display the difference. (2) highlight which metric(s) cause the scores to change. (3) provide some diffify-styled output to help the user understand what's changed between the versions their comparing.

AARON-CLARK commented 1 year ago

In a roundabout way, another person voiced their opinion that that app could potentially tackle since it stores riskmetric scores in its database.

https://github.com/pharmaR/riskmetric/issues/266

Basically, the app could help users make a better informed decision on the package if they see a trend of past risk calculations. That would also help them contextualize the newest score is consistent/ inconsistent with the trend.

Use case: if a user sees a trend of strong risk scores, but then the risk score is higher than the norm, perhaps they shouldn't judge it as harshly. And vice-versa: if the trend shows poor risk scores, and then the latest score is low, they could be more cautious of the package's overall risk.

Somewhat related to #253

AARON-CLARK commented 1 year ago

Regarding a trend of past risk scores, that will likely be more easily tackled once the new riskscore pkg is developed, which would have risk scores for all CRAN pkgs running the latest version of riskmetric. More info here: https://github.com/pharmaR/pharmaR/issues/19

AARON-CLARK commented 1 year ago

Also, I forget if this is logged in another issue, but also need to consider a module that compares two packages against one another! And that comparison would need to be able to output to a report. The use case being, you have two packages that perform the same statistical procedure, you'd want to know which one is "better" based on their risk metrics / assessments & score.

aclark02-arcus commented 3 weeks ago

Per community meeting today, 8/20/24, Novartis presented a few helpful visuals:

Action: Novartis tweaked one requirement / rule for defining "low" risk pkgs from 1k downloads per month, to 50k, so some pkgs didn't have enough downloads and their decisions were changed. Thus, they had to write Performance Qualifications (PQ) tests to resolve the issue & keep them in the bundle.

Action: CRAN archived a few pkgs for more than 180 days, so Novartis removed them from the GxP bundle, since the pkgs need to be on CRAN.