Examples of CSA and filtering

[Stefan-Doering-BI]

Hi @dgkf and @Xyarz , I tried to create two examples (specific intended-use vs broad indended-use) to get a feeling how the ideas from the computer software assurance guideline might work with our filtering feature. This mostly for brainstorming and discussion.

Package: BLRM

Assumed intended-use category: “Specific” Features, Functions, or Operations: function 1: prior; function 2: data … Assumed Intended Use of the Features, Functions or Operations: Provides dose escalation design in Phase I oncology trial using Bayesian Logistic Regression Modeling given prior and observed cohorts. Proposed Risk-Based Analysis: Failure of this package may provide an incorrect estimate for a dose escalation, but would not foreseeably lead to compromised safety because the provided dose escalation design is only considered one of many inputs which are used within the decision making process. Therefore, this package does not pose a high process risk. Proposed Risk Category: “Not high” Proposed Assurance Activities: Acceptance Criteria 1: “Code Coverage >80% ” Acceptance Criteria 2: “Independent review and approval of test plan by Validation Hub available” Acceptance Criteria 3: “Package documentation available”

Package: stats

Assumed Intended-Use Category: “Broad” Feature, Function, or Operations: function 1: t.test Assumed Intended Use of the Features, Functions or Operations: Assumed intended use of this functionality may include primary study end-points … safety critical decisions … Proposed Risk-Based Analysis: Failure of this package may lead to … foreseeably lead to compromised safety … Proposed Risk Category: “High” Proposed Assurance Activities: Acceptance Criteria 1: “Code Coverage >95% ” Acceptance Criteria 2: “Independent review and approval of test plan by Validation Hub available” Acceptance Criteria 3: “Package documentation available” Acceptance Criteria 4: “Assurance activity: Scripted Testing: Robost available”

[dgkf]

My first instinct as a developer is to try to pull as much of this information from existing sources as possible, so I'll just run through some ideas that might help us streamline this categorization:

• Feature, Function or Operations - I think these should be covered in the package manual (ggplot2 example). Do you feel it lacks some critical information?
• Proposed Assurance Activities - Would you see the the risk category as always mapping into some criteria? ie, medium always maps into the first set of criteria, high always maps into the second set of criteria?
• Assumed Intended Use - Would the package title and description suffice for this?
  For example, dplyr describes its purpose as:

  A Grammar of Data Manipulation
  A fast, consistent tool for working with data frame like objects, both in memory and out of memory.

[Stefan-Doering-BI]

• Feature, Function or Operations - I think these should be covered in the package manual (ggplot2 example). Do you feel it lacks some critical information?

For a computerized system it probably wouldn't, but for software package this might suffice. That is maybe a good place to start from.

• Assumed Intended Use - Would the package title and description suffice for this? For example, dplyr describes its purpose as:

  A Grammar of Data Manipulation A fast, consistent tool for working with data frame like objects, both in memory and out of memory.

Often this should be possible and sufficient. That is the motivation behind the 'Assumed intended-use category'. For broad it is hard to make assumptions about it`s specific use and the package description should more or less tell what needs to be said. For 'Specific' you might want to add assumptions to the intended-use. That is exactly why I thought BLRM is a good example to provide. If BLRM would be used as an automation that directly does the decision making. Then I think the package should be high-risk. In practise it is not done this way (at least that this the assumption) which makes it arguably 'Not high-risk'.

• Proposed Assurance Activities - Would you see the the risk category as always mapping into some criteria? ie, medium always maps into the first set of criteria, high always maps into the second set of criteria?

If feel it would be desirable to have set of criteria, the challenge is that I guess we want also some wiggle room and justify for example not performing 'Scripted Testing: Robost', because a package was tested maybe through unscriped testing approaches already or maybe we feel there is enough emperical evidence that makes it very unlikely that the package is not working. But for the MVP maybe the simpliest scenario, which I hope can be expressed in a set of criteria is the way to go.