Dolphin release candidate testing - AAD Product Assessment Sanity Testing

[nanda-katikaneni]

💡 Summary

In preparation of releasing the Dolphin or v0.3.0 of ScubaGear code, conduct sanity testing of AAD product. Objective and Scope of the task are provided below.

Objectives:

1. There are no regression issues in AAD product assessments with Dolphin Release
2. Additional sanity testing to ensure that: each policy assessment result is shown in the report, assessment works against all available tenants (G5/E5, G3/E3)  
3. AAD product assessment works both in interactive and non-interactive (service principal) modes.

Scope:

1. Detailed functional testing of each policy statement result is out of scope
2. Consistent results between interactive/non-interactive modes and no operational issues in running the test against all tenant types are within the scope.

Motivation and context

This would be useful to ensure that Dolphin release is stable

Implementation notes

Before the test, ensure that test user has minimum user role on a given tenant to assess AAD (look into README). Then, execute the AAD product assessment on all available tenants – first in interactive mode and then in non-interactive mode. After the test verify the following:

1. Verify that all tests run without errors and results reports are generated. Each policy has a result (no empty results)
2. Ensure that there are no regressions from Coral release – for the tested tenant compare the result report from current assessment against saved Coral release results - ensure that any different result is consistent with code change (provide a detailed explanation on any observed diff in results)

Acceptance criteria

1. AAD product assessment works in both interactive and non-interactive mode against G5, E5, G3 and E3 tenants.
2. There are no crashes and/or empty results
3. Results are consistent with Coral release assessment results - any diff is consistent with code changes (viz. support for conditional access policies)

[schrolla]

My results from AAD release testing based on the stated objectives. Note that the same testing was performed in the available G5, G3, and E5 environments. In summary, all environments returned the expected results without any errors, crashes, or unexplained program behavior. As such, release testing appears to be successful without any additional bugs or issues to be addressed.

1. There are no regression issues in AAD product assessments with Dolphin Release Ran Invoke-Scuba -p aad first using the ScubaGear v0.2.1 release and then again with the v0.3.0 release candidate using a test account with minimum permissions (Global Reader + required scopes). While some differences were observed in the test results, these were explained by the conditional access policy user/group exclusion capabilities added in the new release where test tenants had such exclusions. A configuration file exempting these exclusions was created and ScubaGear re-ran via Invoke-Scuba -ConfigFilePath aad-config.yaml which produced results matching v0.2.1 in all but AAD 2.13 as it does not support role exclusions by design and the tenant CAP included such an exclusion. This was expected behavior.

2. Additional sanity testing to ensure that: each policy assessment result is shown in the report, assessment works against all available tenants (G5/E5, G3/E3)
   The same tests as shown in part one were performed across the available G5, E5, and E3 test tenant environments. While assessment results varied across tenants due to configuration differences, all results were consistent between the 0.2.1 and 0.3.0 tool versions except where differences were to be expected. No errors were encountered.

3. AAD product assessment works both in interactive and non-interactive (service principal) modes. The same tests were then repeated for v0.3.0, but using non-interactive service principal credentials with the same minimum role and scopes. The results were identical to the interactive results without any errors or issues.

[schrolla]

Testing was completed and reviewed with team.