Application security testing metrics

[mcsnps]

[AIS-05] Automated Application Security Testing Security testing must be enabled:

• at coding development using a plugin,
• before deployment using CI/CD pipeline,
• runtime Data related to testing enabled or not may be gathered and metrics may be generated. This will provide visibility if in any place security testing is not happening that needs to be activated. Alert may be sent if, for any reason, security testing is stopped.

[pritikin]

6/8 team discussion AIS-05 and AIS-06 are tightly linked. We have a current metrics for AIS-06 (where the text might relate to -05). Discussion proposal. Have two metrics.

AIS-05 to focus on the security testing itself (did it happen automatically)

AIS-06 to focus on the lineage.. can we trace the current production code back to the tests that were run on it (this is substantially what the current metric does: "measures the percentage of running production code that can be directly traced back to automated security and quality tests that verify the compliance of each build"). This tells us that tests were run but doesn't tell us what kind of tests. We would , for example, measure 100% even if half the systems did functional tests and the other half did security compliance tests. Provides a measure of "coverage" bound to our lineage conversation.

Another problem: you can have the verification step but ignore the results. Capturing this would be an improvement on the metric. Here we want to provide a measure of "enforcement" or "effectiveness".

Proposed -05 language would be? The goal is to measure what types of tests automatically occurred? Here we want to know which tests were run on which systems. Sounds like we end up with a series of potential metrics: -05-m1 quality tests "A: Total number of pieces of Production Code that have an associated [quality] Step" -05-m2 security tests "A: Total number of pieces of Production Code that have an associated [security] Step" -05-m3 functionality test

Proposed -06 new metric might be: "A: the number of code deployment process runs that had an integrated automated security scan that succeeded. B: all code deployment processes executed" [discussion in meeting: this is substantially the same meaning as the current text] We're broken on -06. It needs to reflect the security of the deployment process.

Summary: current -06 is actually an -05 metric that would benefit from being broken into measuring distinct [quality|security|functionality| tests. We have the basic language for this from above; just need to break it up.

new -06 needs to focus on the security of the deployment process. We need language proposed for this.

[pritikin]

An additional question: what do we do with the current -06 metric? deprecate?

[mcsnps]

In my opinion, AIS-06 primary control description seems to be related security of CI/CD pipeline used for deployment of production code. We can do continuous delivery with manual approval or continuous deployment with automated approval based on results from tests performed in the pipeline. The current AIS-06-M1 does not really reflect that.

From: Max Pritikin @.> Sent: Wednesday, June 8, 2022 12:58 PM To: cloudsecurityalliance/continuous-audit-metrics @.> Cc: Monika Chakraborty @.>; Author @.> Subject: Re: [cloudsecurityalliance/continuous-audit-metrics] Application security testing metrics (Issue #23)

An additional question: what do we do with the current -06 metric? deprecate?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/cloudsecurityalliance/continuous-audit-metrics/issues/23*issuecomment-1150165374__;Iw!!A4F2R9G_pg!YDtgwBlOV5CSpLg0Ac6Vjv3H9toG0l8gfyKF46UgLBJPVkJ7kOuCvCdEHdjK0gR3HHbvys-KsqcUyrgZfjaO6iJo$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AZJ7QAJDEV6BXDMVO5FEYRDVODGLDANCNFSM5WZXEUWQ__;!!A4F2R9G_pg!YDtgwBlOV5CSpLg0Ac6Vjv3H9toG0l8gfyKF46UgLBJPVkJ7kOuCvCdEHdjK0gR3HHbvys-KsqcUyrgZflDZy2SP$. You are receiving this because you authored the thread.Message ID: @.**@.>>

[apannetrat]

Proposal:

• Create 3 metric for AIS-05 -05-m1 quality tests "A: Total number of pieces of Production Code that have an associated [quality] Step" -05-m2 security tests "A: Total number of pieces of Production Code that have an associated [security] Step" -05-m3 functionality test

Create a new AIS-06 metric:

• "A: the number of code deployment process runs that had an integrated automated security scan that succeeded. B: all code deployment processes executed"

For now, we keep the existing AIS-06-M1 metric (we might deprecate it in the future).

Next step: write the YAML and generate PR.