Project Lifecycle: Impact Stage - Acceptance Criteria

[Naomi-Wash]

Comment moved from Project Lifecycle Document Section 3. Stages - Definitions & Expectations - Impact Stage - Acceptance Criteria

To graduate from the Incubation or Growth Stages, or for a new project to join as an Impact Stage project, a project must meet the Growth Stage criteria plus:

Discussion

Do we need to explicitly call out security vulnerability reporting & management process (or is it adequately covered by governance). This could to be more specific around supply chain security (ie looking to the openssf/scorecards/sbom - though that's probably the effort to approve a specific security policy) SECURITY.md? (as we mention other files) This may be expected to begin by growth, but is essential in impact

we should also probably align with OpenSSF Best Practices passing criteria OpenSSF's vulnerability best practices would be beneficial here and there are also templates on this topic in the cncf's contribute.cncf.io under the maintainer section. we should also include Security Audits, perhaps leverage OSTIF to do this? Otherwise partnering with an group to facilitate joint-security reviews (https://github.com/cncf/tag-security/blob/main/assessments/guide/joint-assessment.md) is another start if an audit cannot be funded.

Are there cases where a security audit may not be required in order for a project to be in this stage? If so, perhaps a layer of "security assurance" (just brainstorming a term for the sake of this example, don't read too much into it) abstraction could be the requirement. And then there could be multiple ways of checking the box on that security assurance requirement, an audit (on some periodic ongoing basis?) being one of them.

It depends on what the PQCA wants to do. If the expectation is that adopters (we should define who the target adopters are- other projects, enterprises directly?) can have confidence in integrating Production Impact projects, it is entirely reasonable to expect some form of security review will have occurred. Organizations that ship technology projects do this today - albeit they have security engineers, penetration testers, and plenty of others to do this work (often behind the scenes and not open source) Since many of the contributors to PQCA projects have a specialized field of expertise that does not include security assessments, penetration testing, and auditing, it would be beneficial for the foundation to foot this expense or have other entities "sponsor" the evaluation. there is also another perspective of potentially supporting our projects in seeking FIPS 140-3 certification. (https://csrc.nist.gov/pubs/fips/140-3/final) but this is a much more significant investment and may only be relevant to adopters working in the public sector. Note 140-3 certification, joint-assessments, security reviews, etc. do NOT substitute an independent security audit.

Two points: 1. We can certainly include some security requirements here. 2. The security discussion is a much larger discussion than this, and we should probably have it separately.

This is OK when evaluating software without a security aspect (e.g. "CBOM"), but I strongly urge you to re-consider this stance when looking at crypto libraries such as OQS. OQS at its core is security software. It is software mostly integrated from external sources with very different maintenance levels. It is software mostly developed by cryptographers, not engineers, let alone security software engineers. It's prime mission still is to facilitate experimentation, leading to code being integrated that external users dub "borderline irresponsible". All this leads me to conclude that the "security discussion" is integral to a proper classification of this (security) software and cannot be had separately.

What requirements would you have in mind then? It would be appropriate to require projects to follow "best practices", but those would need to be defined.

Emily has listed quite a few above but the discussion seems to have been ended by your wish to "have it separately". Before reiterating/making further suggestions, isn't the real/key question PQCA needs to answer this: Does PQCA want to create software that is qualitatively as good as a product? The answer may well be "No", e.g. serving companies that want to earn a profit from adding enough bells and whistles to PQCA software to sell/consult around it. It might also be "Yes" by those entities that want to rely on this software as-is or see it as their obligation to strengthen the quality of FOSS. A positive answer to this question would lead to certain requirements becoming essential, e.g., code reviews, FIPS requirements, etc. I'm not in a position to make such decisions. I do have a preference towards qualitatively good FOSS -- but am acutely aware of the obligations and costs this brings along. I'm surely willing to (keep) chipping in my part to the best of my ability -- if others commit to the same.

We do have a PQCA Security workgroup. It hasn't really got up and running yet, but I think that would be the right place for these discussions (multiple comments here could make good issues/discussions there). See https://github.com/PQCA/wg-security In the project lifecycle doc can we just include a statement to say 'adopt the PQCA Security guidelines as a minimum standard' or similar?

That is certainly possible, identify them as a sub category of lifecycle criteria is fine. However ensuring the criteria are all in one place supports maintainers in finding content centrally.

[baentsch]

@Naomi-Wash Can I ask what the plan of action is here? I see all (also my) opinions stated above, but nobody contributing further/leading this to resolution for months now (first in the Google doc, then here). Is there any timeline for adopting the lifecycle document? Is the intention to resolve the above differences in opinion (that are relevant to the lifecycle doc) first? Or is the idea to simply "enact" the lifecycle doc without discussion/getting agreement on its contents such as per this issue?

This obviously also pertains to the proposed change of purpose of PQCA as discussed in #44 and IMO has a profound impact on what PQCA needs to spend its money on (as per the above, e.g., security engineers/engineering, if the mission change occurs) or not (if not).

[brian-jarvis-aws]

Reviewing the correspondence above, I believe it would be a good goal for the security workgroup to define a set of minimum standard security guidelines and best practices that then get referenced from this document. However, independent of that, I think we do need to add a security expectation to the Impact Stage acceptance criteria.

I propose adding the following to the Impact Stage acceptance criteria:

• Explicitly define the project's security policy, vulnerability reporting process, history of vulnerabilities, and history of security audits performed. This is preferably laid out in a SECURITY.md file or other prominent location in the project's codebase and website (if applicable).

[baentsch]

Good improvement, @brian-jarvis-aws . May I suggest to strengthen as follows to move beyond a pure "process focus"

Explicitly define the project's security policy, incl. threat model, vulnerability reporting process, history of vulnerabilities, history of external security audits performed, incl. their goals and results, and external security certifications, e.g., FIPS-140 (if available). This is preferably laid out in a SECURITY.md file or other prominent location in the project's code base and website (if applicable).

[brian-jarvis-aws]

I'm ok with your suggested additions, I just adjusted the wording slightly. Submitted this change in PR #54.