Be Clear on What is Being Audited: Are we Auditing the Management System or the Process of Collecting Metrics

[JonathanChristopherson]

SOC-2 and ISO recognizes two different modes of audit: Are we auditing the management system or the process of collecting metrics.

We need to split these in to two clear processes, and we need to be able to clarify to an auditor that we recognize there are two steps and here's how we recommend going about evaluating them both.

We need to be clear on what the scope is of the audit. Assertion in the conversation is that we are evaluating the management system, but that the management system won't have integrity if we do not understand the process of metric collection.

@mosi-k-platt , @pritikin , and Willie, please update the issue to clarify.

[JonathanChristopherson]

Upstream Impact of this is that different assessors are certified for the two different modes, therefore identifying one mode or the other may prevent an assessor from being able to audit using this automation.

[JonathanChristopherson]

From an SRE/Observability standpoint, how do I gain confidence in the metrics captured to guide me to trust that I am doing the right thing to meet compliance requirements?

[JonathanChristopherson]

We want to avoid overlap with the CCMV4 Auditing Guidelines from that working group. We need to insure we are documenting in the context of what has been produced by our group and avoid clashing with their docs.

We will need to bridge these docs, we will want to write the bridge.

We will also want to consider the "Code of Practice for Implementing and Maintaining Key Metrics" (This document does not yet touch on auditing of the metrics system)

[JonathanChristopherson]

Conversation 2022/03/25: Our focus is to evaluate the management system, but to do that, we need to evaluate the collection system.

We would have two outputs:

• The set of metrics is our output, the current score of the management system, which can be shared with CSA, external customers, and internally within an organization
• The measures.yaml file, which is a data and process security output intended for the auditor to insure that our metrics collection process is accurate

Those two outputs effectively cover both scopes as intertwined.

The evidence files, for instance, wouldn't be shared with CSA, but may be used internally or presented to a third party auditor. What would an evidence file look like?

In presenting the evidence and the logic to the auditor, they will ask "does this number conform to the data I'm seeing, and does this data conform to the number?" In many cases it may just be a question of derived, direct queries, in some cases you may need logical manipulation of the data in order to arrive at the evidence ("interpretation"). One approach is to inject data into the system and see how it responds.

Question: How do we prove to an auditor that the system is working? In the metrics implementation guide they cover this. You don't need to keep a copy of the data, but make the chain of data traceable.

[JonathanChristopherson]

Conversation 2022/03/25: It comes down to what the systems are and how we want to use them. If the objective is to prove the evidence at a point in time, you need to prove that the data captured is immutable. If the point is to demonstrate idempotence of the data, then you don't need a copy of the data but the mechanism used to capture and analyze the data. If you cannot guarantee that the data cannot be changed, you must have a copy of the data. --This should be marked out of scope for this conversation.