Metric proposal: number of controls deployed

[apannetrat]

Two proposals from Walter Williams:

The number of controls that are less than 95% completely deployed in production, along with the rate of change over the last 3 months to show the forecast for when we can expect for each to reach 100%

The number of controls operating within a standard deviation of no more than 6 from the mean of expected behavior to demonstrate the lack of unacceptable adverse impact.

The critical reader will say that those metrics are not against the CCM 4.x. They are and they're not, but they can be meaningfully applied to each and all controls that can be measured on the CCM 4.x, and can also deal with he controls in aggregate to show the effectiveness of the entirety of the system, which I think is a valid way of looking at the CCM from the perspective of how effective are these controls at managing risk.

I will propose more specific metrics shortly.

[apannetrat]

Comment:

• How do you measure the deployment completeness of a control? (i.e. what does it mean that the control is 95% deployed?)

[pritikin]

This kinda reminds me of LOG-10 "Establish and maintain a monitoring and internal reporting capability over the operations of cryptographic, encryption and key management policies, processes, procedures, and controls" which we interpreted in LOG-10-M1 as "measures the percentage of cryptography, encryption and key management controls with defined metrics".

Our framing is that a control is really mature when there is a continuous security metric associated with the control and that metric is automated and actionable. From that perspective a control is "more deployed" on a scale that looks like: from low(er) maturity/deployment

• its in some policy
• there is a defined process
• the process has a metric but the SLO hasn't been achieved
• the process has a metric and the SLO has been achieved
• the information system has a metric but the SLO hasn't been achieved
• the information system has a metric and the SLO has been achieved to high(higher) maturity/deployment

I dislike ordinals although I can see this one working. Perhaps instead we could say:

A: The number of controls with an associated metric that meets or exceeds the recommended SLO B: The total number of defined metrics in this catalog

This would answer Alain's question and provide for a clean metric. It hides a lot of complexity behind the word "associated" which is partially mitigated by B being the number of metrics in the catalog instead of the number of controls. (Recall I reject the idea of a 1:1 mapping between metrics and controls).

[pritikin]

Issue #30 and #39 provide an alternate approach based on existence of configuration baseline tests (rather than logs).

[pritikin]

6/23 Yehia Ahmed points out a couple of other examples:

"AIS-03 Define and implement technical and operational metrics in alignment with business objectives, security requirements, and compliance obligations" " LOG-13 Define, implement and evaluate processes, procedures and technical measures for the reporting of anomalies and failures of the monitoring system and provide immediate notification to the accountable party." with LOG-13-M2 measuring the uptime of the monitoring (metrics) system.