Proposed metrics for AIS-07/TVM-03 and LOG-03/STA-07

[apannetrat]

Some metrics proposed by Walter Williams:

AIS-07/TVM-03

Number of CVSS 7.5 or higher known vulnerabilities with an age of over 30 days CVSS score. Plus Number of CVSS 6.0 through 7.4 known vulnerabilities with an age of over 60 days CVSS Score. Plus Number of CVSS 4.0 through 5.9 known vulnerabilities with an age of over 90 days * CVSS Score.

An acceptable level of risk should be 0, and any value higher than 0 indicates a failure to address known vulnerabilities in a timely manner.

LOG-03/STA-07

The number logs collected from SaaS solutions / the number of SaaS solutions leveraged by the enterprise.

The ratio should be 1, indicating completeness of log collection.

[apannetrat]

Comments:

• The first proposal looks like a special case or variant of the metric AIS-07-M6. Also this metric doesn't seem to consider vulnerabilities that are accepted (e.g. because there are mitigated otherwise).
• The second one needs clarification. Does "logs" refer to "log sources" or "list of log events"? What about SaaS solutions that produce several types of logs? Perhaps we could rephrase the first part of the metric as "The number of SaaS solutions with logging enabled" ?

[apannetrat]

Group discussion:

For AIS-07/TVM-03

• We could have a CVSS oriented metric, with the proposed formula or another one validated by best practice / literature.
• We should consider a CVSS score adjusted to env.

For LOG-03/STA-07:

• Rather than counting the number of SaaS/PaaS/IaaS (whatever) solutions that have logs enable, let's look at how successful we are at logging things.
• Consider using the Mitre attack framework and measure how many are captured in the logs, based on an attack simulation. (e.g. see Splunk solution)

[apannetrat]

Assigned to Yehia A. and Kiran C.

[Complade]

Primary CCMv4 Control ID: LOG-03

Primary Control Description: Identify and monitor security-related events within applications and the underlying infrastructure. Define and implement a system to generate alerts to responsible stakeholders based on such events and corresponding metrics.

Related CCMv4 Control IDs: STA-07

Metric ID: LOG-03-M2

Metric Description: This metric measures the effectiveness of the Cloud Service Provider (CSP) logging and alerting capabilities to catch MITRE att&ck framework vulnerabilities.

Expression: Formula: (1-(A/B))*100 Where A: Number of MITRE att&ck framework vulnerabilities flagged in the CSP provided logs by an att&ck framework simulation tool B: Number of MITRE att&ck framework vulnerabilities flagged in the CSP provided logs.

Rules: The logs must be automatically and systematically exported to a third party MITRE att&ck framework tool and compare simulation results with logs and alerts provided by the CSP.

Sampling period: Monthly

SLO Recommendation: 100%

[Complade]

Primary CCMv4 Control ID: AIS-07

Primary Control Description: Define and implement a process to remediate application security vulnerabilities, automating remediation when possible.

Related CCMv4 Control IDs:

Metric ID: LOG-07-M3

Metric Description: This metric measures the effectiveness of the vulnerabilities remediation process.

Expression: For CVSS 7.5 Formula: (1-(A/B))*100 Where A: The Total Number of Existing CVSS 7.5 or higher known vulnerabilities with an age of over 30 days AND Number of Existing CVSS 6.0 through 7.4 known vulnerabilities with an age of over 60 days AND Number of Existing CVSS 4.0 through 5.9 known vulnerabilities with an age of over 90 days B: The Total Number of CVSS 7.5 or higher known vulnerabilities discovered in the past 30 days AND Number of CVSS 6.0 through 7.4 known vulnerabilities discovered in the past 60 days AND Number of CVSS 4.0 through 5.9 known vulnerabilities discovered in the past 90

Rules: According to the organization policy, some vulnerabilities can be accepted and must not be calculated in this metric. The exception ratio must be measured in a different metric and compared to the organization policy.

Sampling period: Monthly

SLO Recommendation: 100%

[apannetrat]

The discussion around the LOG-03 metric has evolved as follows:

• We need to talk about (security) events rather than logs.
• There is some disagreement in the group regarding the idea of requiring a testing tool that covers a specific framework.
• There is some disagreement about mandating a specific framework such as MITRE as part of the metric.
• Idea: Instead, let's measure our ability to correctly classify existing events against a well-known framework (e.g. MITRE Attack, VERIS DB, NIST) without the need for simulation tools.
• We should probably reclassify this metric under LOG-08 (i/e. "Generate audit records containing relevant security information.")

Sketch of a proposal:

Formula: 100 * A/B A: Number of security events that required action and that where automatically classified correctly under a recognised attack identification framework. B: Number of security events that required action.

Examples of recognised attack identification framework are MITRE Attack, VERIS DB, NIST.

[apannetrat]

Proposals

Test-based approach (LOG-03 or SEF-06)

Description: This metric measures the organization's ability to identify security events by testing attacks from a recognized attack catalog framework such as MITRE Attack, VERIS DB, NIST.

Formula: 100 * A/B

A: Number of security events that were automatically and correctly classified during the test. B: Total number of security events expected to be generated during the test.

Observation-based approach (LOG-03 or SEF-06)

...

Vulnerability remediation (AIS-07/TVM-03)

...

[apannetrat]

The emerging consensus from the working group is that a "LOG-03" based metrics as envisioned above is not feasible:

• In a "test-based" approach, it would require organizations to deploy a SIEM on a test system that is equivalent to their SIEM production setup. This is likely complex, costly and perhaps impractical.
• In an "observational" approach that applies to the production environment, measuring the level of "mis-classification" of security events might not provide a useful metric.