search

cisagov / ScubaGear

Automation to assess the state of your M365 tenant against CISA's baselines
https://www.cisa.gov/resources-tools/services/secure-cloud-business-applications-scuba-project
Creative Commons Zero v1.0 Universal
1.7k stars 222 forks source link

SharePoint functional test plan for the spo variant is failing/erroring for multiple policies #1093

Open nanda-katikaneni opened 6 months ago

nanda-katikaneni commented 6 months ago

🐛 Summary

Ran SharePoint functional test plan for both spo and pnp variants on E5 test tenant. The spo version is failing or erroring for 1.2 and 3.2 policies. The test plan needs to be fixed for clean functional test execution.

To reproduce

Steps to reproduce the behavior:

  1. Run SharePoint functional test suite for spo variant on a E5 tenant
  2. Observe if all complaint and non-complaint test cases are running without error and passing all tests.

Expected behavior

SharePoint test plan in spo variant running without errors and successfully complete all complaint and non-complaint cases.

Any helpful log output or screenshots


Context Execute test, MS.SHAREPOINT.1.2v1 Non-compliant - SharingCapability = ExternalUserSharingOnly (New and existing guests)
VERBOSE: Download the right chromedriver from 'http://chromedriver.chromium.org/downloads'
    [-] Check test case results 10.17s (10.16s|9ms)
     Expected case sensitive like wildcard 'Unknown' to match 'N/A', because unexpected criticality. [Output folder: xxx; Expected: False; Result: N/A; Details: This product does not currently have the capability to check compliance for this policy. See Secure Con
figuration Baseline policy for instructions on manual check], but it did not match.
     at $RowData[2].text | Should -BeLikeExactly "Unknown" -Because "unexpected criticality. [$Msg]", 

```test execution summary
Tests completed in 325.32s
Tests Passed: 43, Failed: 8, Skipped: 0 NotRun: 0
tkol2022 commented 1 month ago

Please drop a comment to describe what this issue is blocked on. Thanks.

nanda-katikaneni commented 1 month ago

SharePoint.1.2.v1 is failing intermittently when setting the tenant with a non-compliant config (specifically for OneDrive external sharing config). Based on the debugging this boiled down to "Set-SPOTenant -DefaultSharingLinkType AnonymousAccess -SharingCapability ExternalUserAndGuestSharing". This test pre-condition is required to be set before the next command is executed; but is failing. Based on an earlier investigation by @mitchelbaker-cisa for a similar issue for pnp variant - issue is boiled down to Microsoft API response. Working with Microsoft team to resolve - until then its blocked.