Compliance percentage incorrect when exempted rules pass

[grismemj]

Summary

When rules that are exempted actually do not fail the test, the reported compliance percentage is incorrect.

Steps to reproduce

Create some rules that will pass, but exempt them, and then run a compliance check. It is easiest to see when things are 100% compliant, and then number of exempt rules does not match the number of failed rules.

Operating System version

I'm testing on 12.6.6, but I'm sure this will happen on all

Intel or Apple Silicon

I'm testing on Apple Silicon, but should happen on Intel too

What is the current bug behavior?

Looks like it is calculating the number of tests passed divided by the number of non-exempt rules, instead of the number of non-exempt rules passed divided by the number of non-exempt rules.

What is the expected correct behavior?

Should calculate the number of non-exempt rules passed divided by the number of non-exempt rules.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

Last compliance scan: Fri Jun 2 15:18:25 EDT 2023

1. View Last Compliance Report
2. Run New Compliance Scan
3. Run Commands to remediate non-compliant settings
4. Exit Enter choice [ 1 - 4 ] 1

Number of tests passed: 115 Number of test FAILED: 16 Number of exempt rules: 18 You are 101.77% percent compliant!

Possible fixes

Change the calculation to number of non-exempt rules passed/number of non-exempt rules

[brodjieski]

@grismemj can you checkout the branch and test the adjustments?

[ZebGris]

Looks good, thanks!

[bernstei]

On a closely related note, when there are exempt rules, the output from the remediate script's report is something like

Number of tests passed: 55
Number of test FAILED: 69
Number of exempt rules: 13
You are 49.55% percent compliant!

This breakdown is then essentially useless, because it does not separate the failed rules into ones that are exempt (and presumably OK by local policy, which is why they are exempt) and ones that are not.

The report would be much more useful in the presence of exempt rules if it gave a but more information on which rules failed.

[BrendaHubbell]

Answer:) To calculate the compliance percentage, we need to determine the total number of tests attempted. This can be done by adding the number of tests passed and the number of tests failed:

Total tests attempted = Number of tests passed + Number of tests failed = 55 + 69 = 124

Next, we need to calculate the number of compliant tests. This can be done by subtracting the number of exempt rules from the number of tests passed:

Number of compliant tests = Number of tests passed - Number of exempt rules = 55 - 13 = 42

Now, we can calculate the compliance percentage:

Compliance percentage = (Number of compliant tests / Total tests attempted) 100 = (42 / 124) 100 ≈ 33.87%

Therefore, the correct compliance percentage is approximately 33.87%.

[brodjieski]

There were adjustments to the compliance calculations made available on a branch tied to this issue (branch is named dev_ventura_issue267). Have you performed testing using this new method? It should account for exemptions on those checks that fail, and give a more accurate percentage.

As for the reporting using the script interactively, it's meant to give a quick snapshot of the status on the system when run locally (typically used during development of your baseline on a test system) Additional details about exempt rules and status of each individual check are available in the logs and corresponding .plist file. These files are available for additional processing to build your own reporting based on the data generated from the scans...whether it's in the form of a Jamf EA, Splunk analysis, or custom script.

[bernstei]

Thanks - I'll take a look at that branch.

[BrendaHubbell]

This issue is related to the usnistgov/macos_security repository and specifically to the calculation of compliance percentages when exempted rules pass.

The problem is that the compliance percentage is currently calculated based on the total number of rules, regardless of whether they are exempted or not. This means that if exempted rules pass, they are still counted as failed rules in the compliance calculation, resulting in an incorrect percentage.

To fix this issue, the compliance calculation should exclude exempted rules from the total count. Only non-exempted rules should be considered when calculating the compliance percentage.

This issue has been reported and is being tracked under Issue #267 in the usnistgov/macos_security repository. Developers are working on a solution to accurately calculate the compliance percentage by considering exempted rules correctly.

[BrendaHubbell]

Me va a tener que pagar!! Lol😜