Proof of Concept - Sharepoint provider MSAL authentication and REST API to get missing fields and uniform API invocation

[twneale]

💡 Summary

Issue #143 identified a way to invoke REST endpoints for Sharepoint / OneDrive fields we cannot get from the current PnP and Sharepoint modules by obtaining an access token from Microsoft using MSAL DLLs. This issue is to code a proof of concept and then present the solution to CISA for a decision on whether or not we should pursue integrating this into the ScubaGear codebase.

The solution developed in this issue would replace the existing dependencies on the PnP and Sharepoint Powershell modules with custom code that calls the Sharepoint APIs directly and authenticates using the MSAL dll (which mimics what cmdlets such as Connect-SPOService do internally).

Motivation

• ScubaGear can have a single way of acquiring the data in the Sharepoint provider. Currently the provider code calls PnP for service principal and Sharepoint for interactive authentication so we end up maintaining a fork in the code.
• We can remove unnecessary logic from the Sharepoint / OneDrive Rego that checks if PnP is being used and outputs a “cannot get this field” message for specific policies.
• The PnP module we depend on is an old version (new versions require Powershell 7) and therefore we don’t get PnP code updates. Even if they added the configuration fields that are currently missing from the PnP module we are using, we wouldn’t be able to leverage them because we are tied to the old version.
• Missing fields
  • We can get the “1.2 – external sharing level” configuration field which is currently missing from the PnP module.
  • We can get the “1.3 - external sharing by security group” configuration field which is currently missing from both the PnP and Sharepoint modules.
  • We can get the “3.2 - file and folder permissions for anyone links” configuration field which is currently missing from the PnP module.
• Reduce multiple interactive authentication prompts
  • User logs in a single time, then ScubaGear passes the token across the providers (AAD, Sharepoint) behind the scenes - no need for the user to select their credentials multiple times. This was proven between MS Graph and Sharepoint REST APIs in the proof of concept - we haven't tried between the other providers such as Defender and PowerPlatform.

Implementation notes

Develop a proof of concept with the following characteristics

• [x] The POC can authenticate to Azure AD and Sharepoint using the MSAL library directly.
• [x] Grab the token generated from Azure AD login and use it to mint an authorization token unique to Sharepoint under the hood (no user prompt).
• [x] Use the Azure AD token to call MS Graph APIs
• [x] Use the Sharepoint token to call the Sharepoint web API directly
• [x] Determine the format of the Sharepoint web API call - what does the request look like?
• [x] Determine the format of the Sharepoint web API response - what fields does it return and in what format? Related issue #958
• [x] POC works for interactive authentication
• [ ] POC works for service principal

[tkol2022]

Met with Thom today and he has completed almost all of the actions documented in the issue opener above. Excellent work! The only pending item is to get the POC working with service principal - currently there is an error related to consent that we need to work through.

Thom answered the questions below:

• Are the PnP and Sharepoint modules both calling this XML API endpoint? https://tenantname-admin.sharepoint.com/_vti_bin/client.svc/ProcessQuery
  • Yes
• Can you verify that the REST API can return JSON instead of XML?
  • Yes, however we still have to send it an XML payload.
• If we can get the JSON from the API, how many changes do we need to make to the existing SharepointConfig.Rego policy rules based on the JSON structure returned from the API? Small changes? Big changes?
  • Looks very similar to response we currently get through the Cmdlets.
• Have you gotten your new authentication code to work when authenticating with a service principal?
  • Not yet. We are getting a consent error in Powershell but not Python.
• To call this API we would need to mint an authorization Bearer token that has specific properties targeted toward the specific endpoint.

[tkol2022]

Once this issue completed, the plan is to meet with CISA to determine if making changes to ScubaGear to acquire the missing fields is feasible.

[tkol2022]

Summary 6/25/2024

I met with Thom today to chart a set of next steps. Here is a summary with action items.

• Thom has taken a portion of MSAL.PS and tailored it to potentially work with ScubaGear. He was able to get interactive auth working (user/pass) and was in the process of non-interactive auth but was getting some unexplained consent warnings. I am confident that we could resolve the consent warnings with some tweaks.
• Overall, if we could get this to work, we would untie our dependency on the old PnP version and we would have a single Sharepoint provider module that could retrieve all the fields in the baseline.
• There are really only a handful of configuration settings (3-4) that cannot currently be consistently pulled by either the PnP or Sharepoint powershell packages. This begs the question, is the work involved to get the custom authentication code that Thom was prototyping working, worth the effort and maintenance? There may be a possibility that we could use a version of his authentication code for ScubaGear overall and reduce the need for users to click on multiple prompts during interactive authentication (which would greatly improve the user experience). At the same time, writing and maintaining Powershell code at the lower level required could open up new headaches for the team that we may not feel are worth the pain. Maintaining the code would also require developers that have an intimate knowledge of M365 authentication/authorization logic which tends to be isolated to a select few.

Action items

• [x] Thom commit custom authentication scripts to private repo.
• [x] Thom document how to run code
• [ ] Thom schedule hands-on demonstration to the team. We can decide if there should be any more work done on the code.
• [ ] Ted review Thom's code along with CISA

[twneale]

Repo: https://github.com/mitre/scubagear-msal-research README: https://github.com/mitre/scubagear-msal-research/blob/master/README.md @tkol2022

[tkol2022]

I verified that setting MS.SHAREPOINT.4.1v1 has been removed from the MS Sharepoint admin portal so we will be removing it from the baseline. That leaves the list of settings that could potentially be retrieved by an alternative way of calling the Sharepoint REST API to a total of 3 settings. Refer to this table in an earlier issue for a list of the remaining policies that we might be able to grab from Sharepoint if we created a custom MSAL authentication and the undocumented Sharepoint XML endpoint.

[tkol2022]

Moving this to next release because I am continuing to test this and had to deprioritize due to functional test fixes and other bugs that came up, as well as deprecated features.

[schrolla]

@tkol2022 Do we want to throw this back into the backlog for the time being? Or is this still reasonable to include in Jellyfish?

[tkol2022]

Notes

I was not able to finish testing and analysis of the code that was developed yet, due to other priorities such as bug fixes and preparing for a big government release. Now that I am resuming, this is simply a place I can keep my most current notes on using the .NET MSAL library to authenticate to M365 REST endpoints.

• One of the limiting factors with interfacing with the .NET MSAL methods using the deprecated MSAL.PS powershell module as a reference is that MSAL.PS relies on versions of Microsoft.Identity.Client and potentially other dependent .NET packages which are three years old. We would need to develop a solution that works with the up-to-date version of Microsoft.Identity.Client so that we can get the bug fixes, enhancements and security fixes that get continuously published as part of the lifecycle of that package.