Evaluate AzureAD Conditional Access Policy conditions more precisely

[schrolla]

AzureAD uses conditional access policies to evaluate access requests. These policies may include references to existing users, groups, and roles with both an inclusion and exclusion condition. As such, evaluating policies against SCB baselines that specify ALL or NONE is both difficult and violates standard practice of having break glass accounts that are highly restricted but do not necessarily have the same protections in place as standard accounts.

If a conditional access policy (CAP) includes exclusions, the conformance automation should evaluate the exclusion and provide feedback to the user about the scope and extent of the exclusion relative to the baseline policy. Possible implementations include, for example, a baseline policy that requires that ALL users have MFA enabled should either fail if an exclusion is present OR pass with warnings and indicate the exclusion as a warning. Alternatively, the automation may allow an agency user to specify acceptable exclusions that would not produce a warning or only produce them if a runtime flag was provided. Either way, CAPs should be evaluated more precisely and additional feedback provided to the user.

[schrolla]

This item was split from #92 which addresses the simpler issues of incomplete conditional access policy evaluation.

[schrolla]

There are a couple different items to keep in mind here:

1. The exclusions are captured in our current conditional access policy (CAP) extractions, but we aren’t processing them in any way. Those exclusions can include users, roles, or groups. Each is listed in its own key under within each specific CAP and is formatted as a JSON array of GUIDs associated with the user/role/group object it references.
2. Because we only have the GUID, we don’t necessary have any additional information about the principals that are excluded in the provider JSON data. So doing any real analysis of which principal it may be might also require provider updates. That said, we don’t want to download mappings for every user, role, and group in AD.
3. First steps may simply be to identify all the AAD policies that utilize CAPs and adjust their tests to flag anytime exclusions are present in the policy.

Beyond that it gets hazier as to how to treat exclusions. If we allow for user input of break glass or exempted principals that should not fail policy tests, then we would need to support users ability to specify those principals (via GUID or Display Name we would need to use to get the GUID), and then incorporate those exemptions into the tests as well.

[schrolla]

Based on review of all the AADConfig_2.xx_test.rego scripts, it seems the following policies utilize CAPs:

• 2.1. Legacy authentication SHALL be blocked • 2.2 Policy 1. Users detected as high risk SHALL be blocked • 2.3. Sign-ins detected as high risk SHALL be blocked • 2.4 policy 1. MFA SHALL be required for all users • 2.9. Sign-in frequency SHALL be configured to 12 hours • 2.10. Browser sessions SHALL not be persistent • 2.13. MFA SHALL be required for user access to highly privileged roles • 2.17. Managed devices SHOULD be required for authentication

Next step is to adjust the tests to flag anytime exclusions are present in the policy.

[schrolla]

Reviewed the file AADConfig.rego (where policy tests are defined) to identify the instances of "input.conditional_access_policies" as that is the key for being inspected by Rego for CAPs. The following eight policies include that key:

• 2.1 policy 1
• 2.2 policy 1
• 2.3 policy 1
• 2.4 policy 1
• 2.9 policy 1
• 2.10 policy 1
• 2.13 policy 1
• 2.17 policy 1

Next step is to try modifying one of the policy tests (e.g., 2.9 as it is most straightforward) to introduce exclusions (e.g., using the "ExcludeUsers" key) and check that the tests can still complete successfully.

[schrolla]

Notes from stand up: Adding these exclusions directly will fail in most real world cases as exclusions for break glass or emergency accounts are common. In the future, one way to handle this is to support an option for users/agencies to specify documented exceptions that would not fail the test, only undocumented exceptions would fail. However, other features must be implemented before we can reasonably accept this input from users.

Until then, we need to determine a way to detect the CAP exclusions (which we have) and modify the result in a way that:

• Tells the user that a policy to meet the requirement exists
• But that said policy is not fully in line with the policy due to specific exclusions

Possible solutions include keeping the color the same, but including additional report details about the policy and its exclusions OR changing the result color to yellow to indicate a partial requirement met or warning about the requirement being met.

[schrolla]

Adding some details in preparation for the meeting. Sorry if this is unclear and will try to clarify at the meeting. Thank you, Shanti

2.1 policy 1: Legacy Authentication SHALL Be Blocked

• Rego policy requirements: This policy applies to all users and applications. No users, groups, or rules are explicitly excluded by the policy.
• Rego policy unit test:. The unit test checks for whether an arbitrary user, group, role is excluded (Rego: “ExcludeUsers": ["8bc7c6ee-39a2-42a5-a31b-f77fb51db652"] Note the test uses the double-negative check: test_ExcludeRoles_Incorrect. Because no users, groups, roles are excluded in the policy, the unit test correctly passes.

2.2 policy 1: Users detected as high risk SHALL be blocked

• Rego policy definition: This policy applies to all users with high user risk levels and all applications. Again, no users, groups, or rules are excluded by the policy.
• Rego policy unit test: The unit test checks for whether an arbitrary user, group, role is excluded. But since none are excluded, the test passes the double negative check.

2.3 policy 1: Sign-ins detected as high risk SHALL be blocked

Rego policy definition: This policy applies to all users with sign-ins detected as high-risk and all applications. No users, groups, or rules are excluded. Rego policy unit test: The unit test checks for whether an arbitrary user, group, role is excluded. But since none are excluded, the test passes the double negative check.

2.4 policy 1: MFA SHALL be required for all users.

Rego policy definition: This policy applies to all users and applications. No users, groups, or rules are excluded. Rego policy unit test: The unit test checks for whether an arbitrary user, group, role is excluded. But since none are excluded, the test passes the double negative check.

2.9 policy 1: Sign-in frequency SHALL be configured to 12 hours.

Rego policy definition: This policy applies to all users and applications. No users, groups, or rules are excluded. Rego policy unit test: The unit test checks for whether an arbitrary user, group, role is excluded. But since none are excluded, the test passes the double negative check.

2.10 policy 1: Browser sessions SHALL not be persistent.

Rego policy definition: This policy applies to all users and applications. No users, groups, or rules are excluded. Rego policy unit test: The unit test checks for whether an arbitrary user, group, role is excluded. But since none are excluded, the test passes the double negative check.

2.13 policy 1: MFA SHALL be required for user access to highly privileged roles.

Rego policy definition: This policy applies to all applications and two privileged roles. Rego policy unit test: The unit test verifies that if the role is other than the defined privileged role, then the test for incorrect_role passes (double negative)

2.17 policy 1: Managed devices SHOULD be required for authentication.

• Rego policy definition: This policy applies to all users and applications with trust settings applied for compliant devices and hybrid Azure AD joined devices. No users, groups, or roles are explicitly excluded.
• Rego policy unit test: Failure of unit test when attempting to specify that no users, groups or roles are excluded (i.e., policy definition in the AADConfig.rego: count(Cap.Conditions.Users.ExcludeUsers) == 0 as was done for 2.1 through 2.10 cases)

Input JSON provider settings are in the ProvidersSettingsExport.json

Example output for 2.1: PS C:...\CISA-SCuBA-M365-SCB-216-cap-policy-exclusions\Testing> .\RunTestCases.ps1 -p AAD -b 1 -v ...Testing 01 .\AAD\AADConfig2_01_test.rego: data.aad.test_Conditions_Correct: PASS (10.1011ms) data.aad.test_IncludeApplications_Incorrect: PASS (0s) @@@@@_data.aad.test_ExcludeUsersIncorrect: PASS (10.0701ms)@@@@@ data.aad.test_IncludeUsers_Incorrect: PASS (0s) data.aad.test_ClientAppTypes_Incorrect: PASS (10.0486ms) data.aad.test_BuiltInControls_Incorrect: PASS (0s) data.aad.test_State_Incorrect: PASS (10.5963ms)

PASS: 7/7

[nanda-katikaneni]

@schrolla and @ssatyapal123 Based on recent changes to Conditional Access Policy Exclusions (done in Dolphin or v0.3.0) and in view of #232 - do we still need this issue?

[schrolla]

@nanda-katikaneni I don't believe this issue is still needed as the user/group exclusions have been addressed. We do still need to address other CAP conditions that can result in false policy matches where the idea is for the policy to apply to "ALL" though. #92 was supposed to be the split, but I'm afraid that might be too narrow. Might be best to close them both out and create a new, focused item to address CAP session and condition values in policy evaluations that could similarly lead to matches that aren't in line with the specified policy item.

[ahuynhMITRE]

Added details from issue #316 to combine similar issues.

AAD 2.1 "Legacy authentication SHALL be blocked" produced a false positive for an agency pilot. The agency has legacy authentication blocked but only for locations outside of the US. For this they configured the "locations" and "client apps" sections within the conditions for a conditional access policy.

Please email Andrew Huynh for the provider json for further investigation.

P.S. the results was found using the Scubagear module version 0.2.1

[tkol2022]

Via testing, I verified that this is still a problem for all of the conditional access policies.

[schrolla]

The latest version of ScubaGear has largely addressed the assignments portion of the original issue by checking and allowing for defined exceptions via config file. What remains is to also check policy conditions (such as location, device, platform, etc) such that narrowly scoped conditional access policies do not render assessment items requiring broadly scoped CAPs as passing.