search

microsoft / Microsoft365DSC

Manages, configures, extracts and monitors Microsoft 365 tenant configurations
https://aka.ms/M365DSC
MIT License
1.59k stars 500 forks source link

Azure Pipelines on DevOps with certificate-based authentication and Graph v2.0 not working #3454

Closed ricmestre closed 1 year ago

ricmestre commented 1 year ago

Details of the scenario you tried and the problem that is occurring

Note: I know this is not a problem with M365Dsc itself, but that's what I'm using in the pipelines and I don't know where else to report the problem.

Currently Windows containers running on Azure Pipelines on DevOps have Graph v1.28 pre-installed, when installing the latest M365Dsc version v1.23.705.1 they get updated to v2.0 (Update-M365DSCDependencies) and whatever is outdated is uninstalled (Uninstall-M365DSCOutdatedDependencies).

Nevertheless calling simple cmdlets such as Get-MgUser or Get-MgGroup ends up in error, see below, I cannot test every single resource to check which ones are affected by this but my solution relies on calling Get-MgGroup to translate GroupIds to DisplayNames and vice-versa due to the way Assignments work on Intune resources (only by GroupId) which makes upgrading not an option, which at the same time makes me also not able to test the other issues I reported with the latest version.

I already tested this unsuccessfully:

Verbose logs showing the problem

##[error]Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'

Suggested solution to the issue

Update Windows container image to have Graph v2.0 by default instead of v1.28.0.

The DSC configuration that is used to reproduce the issue (as detailed as possible)

N/A

The operating system the target node is running

OsName : Microsoft Windows Server 2022 Datacenter OsOperatingSystemSKU : DatacenterServerEdition OsArchitecture : 64-bit WindowsVersion : 2009 WindowsBuildLabEx : 20348.1.amd64fre.fe_release.210507-1500 OsLanguage : en-US OsMuiLanguages : {en-US}

Version of the DSC module that was used ('dev' if using current dev branch)

1.23.705.1

r4hulp commented 1 year ago

@ricmestre Did you have any luck with this? Mine is totally unrelated to M365DSC but my pipeline is broken because of this.

ricmestre commented 1 year ago

@r4hulp No, this is not working yet for me. In terms of M365Dsc I need to keep using version 1.23.628.1 (which uses Graph v1.28.0) in my pipelines until Microsoft updates their containers images to have Graph v2.0 modules installed by default.

ricmestre commented 1 year ago

Just ran a test pipeline 10 times in a row, in one of them I got a container which had Graph v1.28 installed, all other remaining jobs got a container with v2.0 installed so I guess they are gradually updating them right now. It's still not official though, link below shows v1.28.0 as the one installed by default.

https://github.com/actions/runner-images/blob/main/images/win/Windows2022-Readme.md

andikrueger commented 1 year ago

@peombwa do you have any insight on the rollout status of V2.0 in containers or are you aware of a functioning work around?

catalinpopa247 commented 1 year ago

Having the same issue here. My pipeline in Azure DevOps is executing a Powershell script that has Graph commands over a public DevOps agent and almost any Graph command ends up in this error message: ##[error]Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'

I've done various tests, including the ones mentioned: what I can tell you for sure is that as long as Microsoft.Graph v2.0.0 or v1.28.0 is installed, most commands end up with the same error. I've even tried uninstalling the current version that is pushed on the agent (whichever that might be) and installing v1.19.0 - same behavior. Given this last test, I'm inclined to think this isn't a module issue, but some update that went wrong on the DevOps agents. If I corroborate this with the fact that Graph commands work properly locally, it sounds even more feasible.

@andikrueger @peombwa do you guys have any additional insight on this?

ricmestre commented 1 year ago

There seems to be a bigger issue now...

I've run several more tests and Graph v2.0 now is always available, nevertheless just by calling Export-M365DSCConfiguration on Intune workload it fails with:

##[error]The running command stopped because the preference variable "ErrorActionPreference" or common parameter is set to Stop: Function Export-TargetResource cannot be created because function capacity 4096 has been exceeded for this scope.

To "solve" this the value of $MaximumFunctionCount must be increased, I'm able to set it to 32768 (its max value) but for some reason the value gets lost when it starts the export and then I get that error message above which basically means that without solution I'm stuck with running M365Dsc v1.23.628.1, on Azure Pipelines, which is the last one to support Graph v1.0, this is not good and will have to report this to my management.

This can be replicated locally with the following with Graph v2.0 installed:

$MaximumFunctionCount = 4096
Get-Module -ListAvailable Microsoft.Graph.* | `
    Select-Object -ExpandProperty name | `
    Sort-Object -Unique | `
    %{ Import-Module -Force $_ }

@andikrueger Probably you should change the label of this issue to Bug instead of Enhancement.

ricmestre commented 1 year ago

It seems that the first thing that Export-M365DSCConfiguration does is to call $Global:MaximumFunctionCount = 16000 (M365DSCUtil.psm1:L1158, also for New-M365DSCConnection in L1663) but the error message still says that I've set it to 4096? What is going on here?

andikrueger commented 1 year ago

Please have a look at this preliminary solution: https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2148#issuecomment-1634648878

ricmestre commented 1 year ago

@andikrueger This I've just reported is another problem, the issue is still the same since I'm not able to run my pipelines with Graph v2.0 with certificates but I'm not getting the same original problem.

andikrueger commented 1 year ago

Could you try to rollback to AZ.Accounts V2.12.1 within the container, if this would have any impact on these issues?

ricmestre commented 1 year ago

@andikrueger Like I said, this is not the same problem as the original so there's no point in rolling back Az.Accounts, that won't solve my new problem.

ricmestre commented 1 year ago

Additionally I also use AzurePowerShell@5 in my pipeline as @r4hulp so if that was the case I'd have to use PS7 which probably would fix my new problem BUT brings along an whole set of other problems, that's why it's recommended to use M365Dsc with PS5.1 in the first place.

andikrueger commented 1 year ago

Just to make sure, I do get both issues correctly:

  1. function capacity is exceeded during exports
  2. Authentication fails due to “Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'”

switching to PowerShell 7 would not work due to the other restrictions within M365DSC

ricmestre commented 1 year ago

My initial problem was 2. when containers had Graph v1.28 installed, now they have Graph v2.0 installed which is causing me problem 1., but most likely it doesn't affect exports only but affects other use cases as well such as Assert-M365DSCBlueprint, which I'm testing right now in a pipeline, can't say if it affects deployments as well until a little bit later.

ricmestre commented 1 year ago

@andikrueger As expected Assert-M365DSCBlueprint also fails, but this is getting ludicrous, now I've noticed that all the runs I tested which have containers with Graph v2.0 installed cannot install about half of M365Dsc's dependencies through Update-M365DSCDependencies. The first half of dependencies get installed correctly but when it reaches Microsoft.Graph.Beta.Identity.SignIns all others fail

2023-07-13T20:06:21.1349432Z [2023-07-13 20:06:21] - Updating Microsoft365Dsc dependencies
2023-07-13T20:06:21.1399749Z Installing DSCParser version {1.3.0.10}
2023-07-13T20:06:40.9734143Z Installing ExchangeOnlineManagement version {3.2.0}
2023-07-13T20:06:45.7641148Z Installing Microsoft.Graph.Beta.DeviceManagement version {2.0.0}
2023-07-13T20:07:14.3288893Z Installing Microsoft.Graph.Beta.Devices.CorporateManagement version {2.0.0}
2023-07-13T20:07:20.4391276Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:07:20.4401114Z closing the applications.
2023-07-13T20:07:26.1225205Z Installing Microsoft.Graph.Beta.DeviceManagement.Administration version {2.0.0}
2023-07-13T20:07:32.6872658Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:07:32.6886946Z closing the applications.
2023-07-13T20:07:37.1970608Z Installing Microsoft.Graph.Beta.DeviceManagement.Enrollment version {2.0.0}
2023-07-13T20:07:43.1460811Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:07:43.1468116Z closing the applications.
2023-07-13T20:07:45.4320831Z Installing Microsoft.Graph.Beta.Identity.DirectoryManagement version {2.0.0}
2023-07-13T20:08:20.1690475Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:08:20.1697318Z closing the applications.
2023-07-13T20:08:24.8400663Z Installing Microsoft.Graph.Beta.Identity.Governance version {2.0.0}
2023-07-13T20:08:33.0648725Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:08:33.0655369Z closing the applications.
2023-07-13T20:09:02.4197350Z Installing Microsoft.Graph.Beta.Identity.SignIns version {2.0.0}
2023-07-13T20:09:09.0260415Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:09:09.0266726Z closing the applications.
2023-07-13T20:09:15.6410743Z Could not update {Microsoft.Graph.Beta.Identity.SignIns}
2023-07-13T20:09:15.6965405Z Installing Microsoft.Graph.Beta.Teams version {2.0.0}
2023-07-13T20:09:23.3299717Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:09:23.3306468Z closing the applications.
2023-07-13T20:09:42.2736582Z Could not update {Microsoft.Graph.Beta.Teams}
2023-07-13T20:09:42.2737061Z Installing Microsoft.Graph.Beta.DirectoryObjects version {2.0.0}
2023-07-13T20:09:47.3763883Z WARNING: The version '2.0.0' of module 'Microsoft.Graph.Authentication' is currently in use. Retry the operation after 
2023-07-13T20:09:47.3770973Z closing the applications.
2023-07-13T20:09:48.9785769Z Could not update {Microsoft.Graph.Beta.DirectoryObjects}
2023-07-13T20:09:49.4238025Z Installing Microsoft.PowerApps.Administration.PowerShell version {2.0.167}
2023-07-13T20:09:54.4059971Z WARNING: The names of some imported commands from the module 'Microsoft.PowerApps.AuthModule' include unapproved verbs 
2023-07-13T20:09:54.4070426Z that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again
2023-07-13T20:09:54.4073873Z  with the Verbose parameter. For a list of approved verbs, type Get-Verb.
2023-07-13T20:09:54.4238566Z Could not update {Microsoft.PowerApps.Administration.PowerShell}
2023-07-13T20:09:54.4806393Z Installing MicrosoftTeams version {5.3.0}
2023-07-13T20:10:17.4465965Z Could not update {MicrosoftTeams}
2023-07-13T20:10:17.4467837Z Installing MSCloudLoginAssistant version {1.0.116}
2023-07-13T20:10:21.9806395Z Could not update {MSCloudLoginAssistant}
2023-07-13T20:10:21.9806913Z Installing PnP.PowerShell version {1.12.0}
2023-07-13T20:10:33.6043906Z Installing ReverseDSC version {2.0.0.16}
2023-07-13T20:10:38.5298594Z Could not update {ReverseDSC}
ricmestre commented 1 year ago

@andikrueger Deployments with Graph v2.0 for the Intune policies I have are not affected by "1. function capacity is exceeded", and the reason they didn't fail because of missing installed dependencies is because those policies are not using any of the affected modules that cannot be installed as shown above.

This also means that even if function capacity is resolved somehow the export, assert, but also the code I have on my side, will fail because they all rely on calling Get-M365DSCExportContentForResource which calls Get-DSCBlock from ReverseDSC which cannot be installed (the last one from the list above).

Everywhere I look there are new problems.

andikrueger commented 1 year ago

The warning messages above should not be blocking. This happens, if the right module version is present within the system. The Update-M365DSCDependency function needs some updating in the handling of modules with the right version installed. What I'm really concerned about are these messages:

2023-07-13T20:09:54.4806393Z Installing MicrosoftTeams version {5.3.0}
2023-07-13T20:10:17.4465965Z Could not update {MicrosoftTeams}
2023-07-13T20:10:17.4467837Z Installing MSCloudLoginAssistant version {1.0.116}
2023-07-13T20:10:21.9806395Z Could not update {MSCloudLoginAssistant}
2023-07-13T20:10:21.9806913Z Installing PnP.PowerShell version {1.12.0}
2023-07-13T20:10:33.6043906Z Installing ReverseDSC version {2.0.0.16}
2023-07-13T20:10:38.5298594Z Could not update {ReverseDSC}

Could you add the following code:

Update-M365DSCDependencies -Force
$dependencies = Update-M365DSCDependencies -ValidateOnly

foreach ($dependency in $dependencies){

Write-Host "Failed to find $($dependency.ModuleName)"

}
ricmestre commented 1 year ago

@andikrueger It seems that all dependencies are installed after all even if it says "Could not update {...}" for some of them, which means it's failing here since it's the only thing that it's called after Install-Module and that can throw an exception: https://github.com/microsoft/Microsoft365DSC/blob/d0d0592654bc1c6304ef883b03f548011222cf68/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1#L2783

The catch block could use some love and actually show why it failed:

https://github.com/microsoft/Microsoft365DSC/blob/d0d0592654bc1c6304ef883b03f548011222cf68/Modules/Microsoft365DSC/Modules/M365DSCUtil.psm1#L2798

Let's hope this specific problem doesn't cause any other issues, such as the ones I mentioned in previous comment, but in any case, I'm still affected with "function capacity is exceeded", how can I solve this if increasing the value doesn't do anything?

ricmestre commented 1 year ago

This is the current state of affairs, I need some sleep before looking at this again in a few hours.

Problem 1:

Function capacity exceeded (4096 is default value for $global:MaximumFunctionCount in PS5.1, this doesn't apply to PS7.x)

"Solution" to Problem 1:

Copied M365DSCReverse.psm1 and M365DSCUtil.psm1 to my repo, changed them to display $global:MaximumFunctionCount after it's set to 16000 in all places (and it does display 16000 a couple of times in the logs). I'm not joking, this actually makes me go further with the export, but see below.

Problem 2:

With "solution" to Problem 1 in place I get again 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])' before export begins.

"Solution" to Problem 2:

Setup pipeline task AzurePowerShell@5 with azurePowerShellVersion = 9.2.0 (Uses earlier version of Az.Accounts not affected by this problem).

Problem 3 - Part 1:

With "solution" to Problem 2 in place the export now runs but it fails to extract all of my policies and at the end shows another error You cannot call a method on a null-valued expression., but not sure exactly where this one fails.

"Solution" to Problem 3 - Part 1:

In order to see what gets written in the registry about the failed extraction of the policies I modified Start-M365DSCConfigurationExtract in M365DSCReverse.psm1 so that it returns early, before the function fails.

Problem 3 - Part 2:

Outputting 16000 in the screen (just by calling $global:MaximumFunctionCount) seems to be affecting Start-M365DSCConfigurationExtract since it cannot determine the ConnectionMode.

{ Cannot process argument transformation on parameter 'ConnectionMode'. Cannot convert value to type System.String. } \ at Export-TargetResource, C:\Program Files\WindowsPowerShell\Modules\Microsoft365Dsc\1.23.712.1\DSCResources\MSFT_IntuneAppProtectionPolicyAndroid\MSFT_IntuneAppProtectionPolicyAndroid.psm1: line 1097 \ at Start-M365DSCConfigurationExtract, C:\Program Files\WindowsPowerShell\Modules\Microsoft365Dsc\1.23.712.1\modules\M365DSCReverse.psm1: line 624 \ at Export-M365DSCConfiguration, C:\Program Files\WindowsPowerShell\Modules\Microsoft365Dsc\1.23.712.1\modules\M365DSCUtil.psm1: line 1292 \ at , D:\a\1\s\Source\Scripts\Exporter.ps1: line 100 \ at , D:\a_temp\9a1e4fab-0324-4033-9578-a13c871de75f.ps1: line 37 \ at , : line 1

"Solution" to Problem 3 - Part 2:

Display $global:MaximumFunctionCount value but using Write-Host cmdlet, this makes the policies extract now but they all show 16000 at the end of the line before the green checkmark.

[57/64] Extracting [IntuneWifiConfigurationPolicyIOS] using {CertificateThumbprint}...16000 ✅ [58/64] Extracting [IntuneWifiConfigurationPolicyMacOS] using {CertificateThumbprint}...16000 ✅ [59/64] Extracting [IntuneWifiConfigurationPolicyWindows10] using {CertificateThumbprint}...16000 ✅

andikrueger commented 1 year ago

@ricmestre Thank you so much for the detailed summary of the various issue within the pipeline. I just got word, there is a new Graph PS available: V 2.1.0

https://github.com/microsoftgraph/msgraph-sdk-powershell/releases/tag/2.1.0

I just updated the modules within the PR https://github.com/microsoft/Microsoft365DSC/pull/3474 and added the suggestions changes of yours to the output of MaxFunction count and the Update-M365DSCDependencies function

Any chance to test-drive this version?

andikrueger commented 1 year ago

Just found the issue that was causing Update-M§65DSCDependencies to fail.

Error-Mesage: Function Get-MgUser cannot be created because function capacity 4096 has been exceeded for this scope.
shahidsayyed892 commented 1 year ago 
$context = Get-AzContext
$graphToken = [Microsoft.Azure.Commands.Common.Authentication.AzureSession]::Instance.AuthenticationFactory.Authenticate($context.Account, $context.Environment, $context.Tenant.Id.ToString(), $null, [Microsoft.Azure.Commands.Common.Authentication.ShowDialog]::Never, $null, "https://graph.microsoft.com").AccessToken
$secureAccessToken = $graphToken | ConvertTo-SecureString -AsPlainText -Force
Connect-MgGraph -AccessToken $secureAccessToken
Get-InstalledModule Microsoft.Graph

$UserReadAllId = Find-MgGraphPermission User.Read.All -ExactMatch -PermissionType Application | Select-Object -ExpandProperty Id
write-host $UserReadAllId
Invoke-MgGraphRequest : Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'.
At C:\Program Files\WindowsPowerShell\Modules\Microsoft.Graph.Authentication\2.0.0\custom\common\Permissions.ps1:37 char:27
+ ... estResult = Invoke-MgGraphRequest -method GET -OutputType PSObject $_ ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Invoke-MgGraphRequest], MissingMethodException
    + FullyQualifiedErrorId : NotSpecified,Microsoft.Graph.PowerShell.Authentication.Cmdlets.InvokeMgGraphRequest

Script stack trace:
at Permissions_GetPermissionsData, C:\Program Files\WindowsPowerShell\Modules\Microsoft.Graph.Authentication\2.0.0\custom\common\Permissions.ps1: line 37
at Find-MgGraphPermission<Begin>, C:\Program Files\WindowsPowerShell\Modules\Microsoft.Graph.Authentication\2.0.0\custom\Find-MgGraphPermission.ps1: line 202

Facing this issue from 12 July 23. Seems like microsoft has upgraded mggraph to 2.0.0 in devops pipeline, which earlier was 1.28.0

My use case involves calling below commands in azure devops pipeline to automate tasks:

andikrueger commented 1 year ago

@shahidsayyed892 is your use case related to microsoft365dsc?

shahidsayyed892 commented 1 year ago

@andikrueger no, it is related to interaction with Azure active directory.

andikrueger commented 1 year ago

Thanks for the update. I’ve seen you raised an issue within the graph SDK repository. https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2161

I’ll hide the comments above to have a better focus on the M365DSC use case and troubles. Hope this is ok with you. Thanks!

ricmestre commented 1 year ago

@andikrueger No joy, I've added your changes and Graph v2.1 gets installed but the capacity problem came back again. In fact I tried to change back your Write-Verbose to Write-Host as I had before and I still got the same problem so this hackjob doesn't even work anymore.

I'll need to test this back with Graph v2.0 and downgrade Az to 9.2.0.

andikrueger commented 1 year ago

😤 I was really hoping for this change to make a difference and am running out of ideas.

Is there a chance that there was a change in how Graph V2+ exports functions? This has not been an issue with the pre v2 versions.

ricmestre commented 1 year ago

@andikrueger Sorry for not getting back earlier, but I was making a myriad of testing and this just looks like something that was bound to happen at some point in time and only got discovered with the update to Graph v2. This function capacity issue is also something that only happens with PS5 because it was already solved in PS7, in fact $global:MaximumFunctionCount doesn't even exist there.

The table below showing the tests that work still shows that several modules cannot be updated (this is due to not being able to import them due to the capacity issue) but the export still works as expected.

So the shortest fix here would be to add $global:MaximumFunctionCount at the start of Start-M365DSCConfigurationExtract in M365DSCReverse.psm1, but this only works with combination of Graph v2.0 and Az 9.2.0 (which has Az.Accounts < 2.12.2).

As soon as I try the same fix with Graph v2.1 and Az 9.3.0 the function capacity issue comes back again and I don't know what else to test here to make it work.

Works:

1) Graph v2.0, Az 9.2.0, Write-Host -Object $global:MaximumFunctionCount in M365DSCReverse.psm1/M365DSCUtil.psm1

2) Graph v2.0, Az 9.2.0, Write-Host -Object $global:MaximumFunctionCount in M365DSCReverse.psm1

3) Graph v2.0, Az 9.2.0, $global:MaximumFunctionCount = 16000 before Get-M365DSCAllResources in M365DSCReverse.psm1

4) Graph v2.0, Az 9.2.0, $global:MaximumFunctionCount = 16000 at the start of Start-M365DSCConfigurationExtract in M365DSCReverse.psm1

Doesn't work:

1) Graph v2.1, Az 9.3.0, Write-Verbose sentence with $global:MaximumFunctionCount in M365DSCReverse.psm1

2) Graph v2.1, Az 9.3.0, Write-Verbose sentence with $global:MaximumFunctionCount in M365DSCReverse.psm1/M365DSCUtil.psm1work

3) Graph v2.0, Az 9.2.0, Write-Verbose sentence with $global:MaximumFunctionCount in M365DSCReverse.psm1

4) Graph v2.0, Az 9.2.0, Write-Verbose sentence with $global:MaximumFunctionCount in M365DSCReverse.psm1/M365DSCUtil.psm1

5) Graph v2.1, Az 9.3.0, Write-Host -Object "$($global:MaximumFunctionCount)" in M365DSCReverse.psm1

6) Graph v2.1, Az 9.3.0, Write-Host -Object "$($global:MaximumFunctionCount)" in M365DSCReverse.psm1/M365DSCUtil.psm1

7) Graph v2.0, Az 9.2.0, Write-Host -Object "$($global:MaximumFunctionCount)" in M365DSCReverse.psm1

8) Graph v2.0, Az 9.2.0, Write-Host -Object "$($global:MaximumFunctionCount)" in M365DSCReverse.psm1/M365DSCUtil.psm1

9) Graph v2.1, Az 9.3.0, $global:MaximumFunctionCount = 16000 before Get-M365DSCAllResources in M365DSCReverse.psm1
ricmestre commented 1 year ago

Oh and issue https://github.com/microsoft/Microsoft365DSC/issues/3476 was also raised a couple of hours ago with same function capacity issue but this time with AAD workload, Credentials, Graph v2.0 and running locally so it's failing even on a completely different scenario from mine

andikrueger commented 1 year ago

Very well put. I came across the max function count thing yesterday evening while trying to get ahold of the issues described. Indeed it’s solved with PS 7. I’m still worrying about the fact, that we did not change the count of cmdLet used in M365. We switched for another Graph PowerShell Version and this raised the issues as seen by you and others. Something must have changed on the graph ps side.

andikrueger commented 1 year ago

I just created this function to get more insights about the function count for the used MS Graph modules: https://gist.github.com/andikrueger/0336080022bc6d43d4bb3446e8040109

It's not halfway through all PS 2.1.0 modules and the maxfunctioncount will cause an issue.

@ricmestre could you run the script within your pipeline and share the result?

ricmestre commented 1 year ago

@andikrueger oh sorry I'm not home right now and can't test it right now, I'll share the results in a couple of hours

ricmestre commented 1 year ago

@andikrueger Steps below have been made on a pipeline with an AzurePowerShell@5 task which brings along Az 9.3.0 and the container also has Graph v2.0 installed by default. 

2023-07-14T18:56:28.0255028Z Trying to import Microsoft.Graph.Applications
2023-07-14T18:56:28.0480284Z Module function count 265
2023-07-14T18:56:28.0907987Z Function Count 500
2023-07-14T18:56:31.9089806Z Trying to import Microsoft.Graph.Authentication
2023-07-14T18:56:31.9119101Z Module function count 2
2023-07-14T18:56:31.9542967Z Function Count 502
2023-07-14T18:56:35.4621468Z Trying to import Microsoft.Graph.Beta.DeviceManagement
2023-07-14T18:56:35.6200440Z Module function count 947
2023-07-14T18:56:35.9496319Z Function Count 1449
2023-07-14T18:56:48.7497357Z Trying to import Microsoft.Graph.Beta.DeviceManagement.Administration
2023-07-14T18:56:48.7885354Z Module function count 344
2023-07-14T18:56:49.2949088Z Function Count 1793
2023-07-14T18:56:54.3226036Z Trying to import Microsoft.Graph.Beta.DeviceManagement.Enrollment
2023-07-14T18:56:54.3431925Z Module function count 190
2023-07-14T18:56:54.9520562Z Function Count 1983
2023-07-14T18:57:00.1556952Z Trying to import Microsoft.Graph.Beta.Devices.CorporateManagement
2023-07-14T18:57:00.2067658Z Module function count 453
2023-07-14T18:57:01.3607053Z Function Count 2436
2023-07-14T18:57:07.3482530Z Trying to import Microsoft.Graph.Beta.DirectoryObjects
2023-07-14T18:57:07.3639947Z Module function count 13
2023-07-14T18:57:08.2989317Z Function Count 2449
2023-07-14T18:57:12.6193554Z Trying to import Microsoft.Graph.Beta.Identity.DirectoryManagement
2023-07-14T18:57:12.6822897Z Module function count 518
2023-07-14T18:57:14.0736784Z Function Count 2967
2023-07-14T18:57:19.9126839Z Trying to import Microsoft.Graph.Beta.Identity.Governance
2023-07-14T18:57:20.1648318Z Module function count 1223
2023-07-14T18:57:22.9668642Z Function Count 4190
2023-07-14T18:57:43.5908030Z Trying to import Microsoft.Graph.Beta.Identity.SignIns
2023-07-14T18:57:43.6696934Z Module function count 563
2023-07-14T18:57:47.3496143Z Function Count 4753
2023-07-14T18:57:56.7270875Z Trying to import Microsoft.Graph.Beta.Teams
2023-07-14T18:57:56.8399055Z Module function count 757
2023-07-14T18:58:01.8107140Z Function Count 5510
2023-07-14T18:58:18.4760470Z Trying to import Microsoft.Graph.DeviceManagement.Administration
2023-07-14T18:58:18.5110792Z Module function count 81
2023-07-14T18:58:23.6061895Z Function Count 5591
2023-07-14T18:58:32.2180845Z Trying to import Microsoft.Graph.Groups
2023-07-14T18:58:32.2672148Z Module function count 336
2023-07-14T18:58:38.0246824Z Function Count 5927
2023-07-14T18:58:49.5092524Z Trying to import Microsoft.Graph.Planner
2023-07-14T18:58:49.5409715Z Module function count 46
2023-07-14T18:58:55.3223986Z Function Count 5973
2023-07-14T18:59:04.5785448Z Trying to import Microsoft.Graph.Users
2023-07-14T18:59:04.6128947Z Module function count 132
2023-07-14T18:59:10.7080907Z Function Count 6105
2023-07-14T18:59:20.9506689Z Trying to import Microsoft.Graph.Users.Actions
2023-07-14T18:59:20.9874997Z Module function count 159
2023-07-14T18:59:27.3498757Z Function Count 6264
andikrueger commented 1 year ago

This is very interesting. There are 244 functions prior to loading the graph dependencies within the session. After the load of all graph modules, this count is at 6264 - which would still be below the current limit of 10.000. either the other modules add so many other functions or something else must go very wrong.

Would you mind running a final test, to see if the limit of 32767 would resolve all issues?

ricmestre commented 1 year ago

The max value is actually 32768 :)

I'm currently testing using Graph v2.1 and setting the limit to 32768 at the beginning of Start-M365DSCConfigurationExtract, I'll let you know of the results in a bit, but are you thinking in any other tests at this point? After all this I really don't mind making further testing.

ricmestre commented 1 year ago

Oh wait the results are actually not quite right, I messed up! Let me just finish this test which most likely solves this, but I really want to know the real values here. Unfortunately I'm not using pipeline caching of any sorts so this always takes a really long time to see the results.

ricmestre commented 1 year ago

Nevermind, actually it was correct I was just looking at a different pipeline :\

andikrueger commented 1 year ago

Thank you so much for all the legwork you do with this issue. I’m running out of ideas at the moment. I hope this setting will resolve it for now. 👍

ricmestre commented 1 year ago

Nope, it also fails unfortunately. Something is really messed up on Graph's side, v2.1 that is.

andikrueger commented 1 year ago

I’ll need to look into the export function a bit more. I’m wondering if there is something wrong with the way the modules and functions are pre-loaded into the export session. Each call into the resources will load the dsc modules and functions as well…

I hope I get a chance to do so by Monday

ricmestre commented 1 year ago

Yeah, let's call it a day and enjoy our much deserved weekend. Thank you very much to keeping up with me :) Have a nice one Andy!

andikrueger commented 1 year ago

@ricmestre Could you share a bit of your pipeline? I would try to run it within one of my repos.

ricmestre commented 1 year ago

@andikrueger Hi, I have several pipelines, this is the one I use for exporting the blueprint, it's not complete but it's very similar and can help you testing as well, the functions used are very similar as well as to what it's included in the Whitepaper.

You'll just need to update the variables at the bottom of the code for your environment.

The pipeline that you can call it Exporter.yml is first then follows the code for Exporter.ps1.

trigger: none

pool:
  vmImage: windows-latest

steps:
- task: AzurePowerShell@5
  displayName: 'Run Exporter script'
  inputs:
    #azurePowerShellVersion: latestVersion # This will test Az 9.3.0 which causes issues with Graph v2.0 so you'll need to update to v2.1 in the code
    azurePowerShellVersion: 9.2.0 # By using this previous version then Graph v2.0 looks like it works
    azureSubscription: KeyVaultConnection
    scriptType: 'filePath'
    scriptPath: 'Exporter.ps1'
    pwsh: false
    failOnStandardError: true
    errorActionPreference: 'stop'
function Write-Log
{
    [CmdletBinding()]
    [OutputType($NULL)]
    Param(
        [Parameter(Mandatory)]
        [String]$Message,
        [Parameter()]
        [Int32]$Level = 0,
        [Parameter()]
        [Switch]$NoNewLine
    )

    $Timestamp = Get-Date -UFormat "%Y-%m-%d %H:%M:%S"
    $Indentation = "  " * $Level
    $Output = "[{0}] - {1}{2}" -f $Timestamp, $Indentation, $Message

    $Params = @{
        Object = $Output
    }
    if ($NoNewLine) {
        $Params.NoNewline = $true
    }

    Write-Host @Params
}

function Initialize-PSRepository
{
    [CmdletBinding()]
    [OutputType($NULL)]
    Param()

    Write-Log -Message "Configuring PowerShell Gallery"
    Set-PSRepository -Name "PSGallery" -InstallationPolicy "Trusted"
    [Net.ServicePointManager]::SecurityProtocol = `
        [Net.SecurityProtocolType]::Tls12
    $PSGetModule = Get-Module -Name PowerShellGet -ListAvailable | `
        Select-Object -First 1
    if ([String]::IsNullOrEmpty($PSGetModule) `
        -or $PSGetModule.Version -lt [Version]"2.2.4.0") {
        Write-Log -Message "Installing PowerShellGet"
        try {
            Install-Module PowerShellGet -Scope AllUsers -SkipPublisherCheck `
                -Force | Out-Null
        }
        catch {
            $Message = "Error installing module PowerShellGet: {0}" -f $_
            Write-Log -Message $Message
            exit 1
        }
    }
}

function Install-M365DscAndDependencies
{
    [CmdletBinding()]
    [OutputType($NULL)]
    Param(
        [Parameter(Mandatory)]
        [Version]$M365DscVersion
    )

    Initialize-PSRepository

    $Message = "Checking for presence of Microsoft365Dsc module and all " + `
        "required modules"
    Write-Log -Message $Message
    Write-Log -Message " "

    Write-Log -Message "Checking Microsoft365Dsc version"
    $PSGalleryVersion = $M365DscVersion
    $LocalModule = Get-Module "Microsoft365Dsc" -ListAvailable
    $Message = "Required version: {0}" -f $PSGalleryVersion
    Write-Log -Message $Message
    $Message = "Installed version: "
    if (![String]::IsNullOrEmpty($LocalModule.Version)) {
        $Message += "{0}" -f $LocalModule.Version
    } else {
        $Message += "None"
    }
    Write-Log -Message $Message

    if ($LocalModule.Version -ne $PSGalleryVersion) {
        if (![String]::IsNullOrEmpty($LocalModule)) {
            $Message = "Incorrect version installed. Removing current " + `
                "module."
            Write-Log -Message $Message
            Write-Log -Message "Removing Microsoft365Dsc"
            $ModulesPath = "C:\Program Files\WindowsPowerShell\Modules"
            $M365ModulePath = Join-Path -Path $ModulesPath `
                -ChildPath "Microsoft365DSC"
            Remove-Item -Path $M365ModulePath -Force -Recurse `
                -ErrorAction "SilentlyContinue"
        }

        Write-Log -Message "Installing Microsoft365Dsc"
        $CheckVersion = Find-Module -Name "Microsoft365Dsc" `
            -RequiredVersion $PSGalleryVersion -ErrorAction "SilentlyContinue"
        $Parameters = @{
            Name            = "Microsoft365Dsc"
            Scope           = "AllUsers"
        }
        if (![String]::IsNullOrEmpty($CheckVersion)) {
            $Parameters.Add("RequiredVersion", $PSGalleryVersion)
        } else {
            $Message = "Version {0} not found, installing latest version" `
                -f $PSGalleryVersion
            Write-Log -Message $Message
        }
        try {
            Install-Module @Parameters | Out-Null
        }
        catch {
            $Message = "Error installing module Microsoft365Dsc: {0}" -f $_
            Write-Log -Message $Message
            exit 1
        }

        Write-Log -Message "Uninstalling Microsoft365Dsc outdated dependencies"
        try {
            Uninstall-M365DSCOutdatedDependencies | Out-Null
        }
        catch {
            $Message = "Could not uninstall Microsoft365Dsc outdated " + `
                "dependencies"
            Write-Log -Message $Message
            exit 1
        }

        Write-Log -Message "Updating Microsoft365Dsc dependencies"
        try {
            Update-M365DSCDependencies | Out-Null
        }
        catch {
            Write-Log -Message "Could not update Microsoft365Dsc dependencies"
            exit 1
        }
    } else {
        Write-Log -Message "Correct version installed, continuing"
    }
}

function Import-PfxCertificateFromKV
{
    [CmdletBinding()]
    [OutputType([SecureString])]
    Param(
        [CmdletBinding(DefaultParameterSetName = 'Workload')]
        [Parameter(Mandatory, ParameterSetName = 'Workload')]
        [String]$DomainShortName,
        [Parameter(Mandatory, ParameterSetName = 'Workload')]
        [String]$Workload,
        [Parameter(Mandatory, ParameterSetName = 'LCM')]
        [Switch]$LCM
    )

    if ($PSCmdlet.ParameterSetName -eq 'Workload') {
        $KVSecretName = "CertPw-{0}-{1}" -f $DomainShortName, $Workload
        $KVCertName = "Cert-{0}-{1}" -f $DomainShortName, $Workload
    } else {
        $KVSecretName = 'CertPw-LCM'
        $KVCertName = 'Cert-LCM'
    }
    $CertTempFile = "{0}.pfx" -f $KVCertName
    $CertTempPath = Join-Path -Path $env:TEMP -ChildPath $CertTempFile

    $Message = "Retrieving secret {0} from KeyVault {1}" -f $KVSecretName,
        $env:KV
    Write-Log -Message $Message
    $Password = Get-AzKeyVaultSecret -VaultName $env:KV -Name $KVSecretName `
        -AsPlainText | ConvertTo-SecureString -AsPlainText -Force

    $Message = "Retrieving certificate {0} from KeyVault {1}" -f $KVCertName,
        $env:KV
    Write-Log -Message $Message
    $KVCertBase64 = Get-AzKeyVaultSecret -VaultName $env:KV `
        -Name $KVCertName -AsPlainText
    if ([String]::IsNullOrEmpty($KVCertBase64)) {
        if ($PSCmdlet.ParameterSetName -eq 'LCM') {
            return $Password
        } else {
            $Message = "Could not retrieve certificate {0} from KeyVault {1}" `
                -f $KVCertName, $env:KV
            throw $Message
        }
    }

    $KVSecretBytes = [Convert]::FromBase64String($KVCertBase64)

    $Message = "Exporting certificate {0} to temp cert file {1}" -f $KVCertName,
        $CertTempFile
    Write-Log -Message $Message
    $Parameters = @{
        Path  = $CertTempPath
        Value = $KVSecretBytes
    }
    if ($PSVersionTable.PSVersion.Major -gt 5) {
        $Parameters.Add("AsByteStream", [Switch]$true)
    } else {
        $Parameters.Add("Encoding", "Byte")
    }
    try {
        Set-Content @Parameters
    }
    catch {
        throw $_
    }

    $Message = "Importing certificate {0} to store Cert:\LocalMachine\My" `
        -f $CertTempPath
    Write-Log -Message $Message
    $Parameters = @{
        FilePath          = $CertTempPath
        CertStoreLocation = "Cert:\LocalMachine\My"
        Password          = $Password
    }
    try {
        Import-PfxCertificate @Parameters | Out-Null
    }
    catch {
        throw $_
    }

    $Message = "Removing temp cert file {0}" -f $CertTempPath
    Write-Log -Message $Message
    try {
        Remove-Item -Path $CertTempPath -Force
    }
    catch {
        throw $_
    }
}

$Microsoft365Dsc = "Change M365Dsc version here"

$TenantId = "FQDN of tenant"

$DomainShortName = "Name of domain without .onmicrosoft.com"

$WorkloadName = "e.g. Intune"

$ApplicationId = "AppId used for this workload"

$CertThumbprint = "Certificate Thumbprint for this workload"

Install-M365DscAndDependencies -M365DscVersion $Microsoft365Dsc

Import-PfxCertificateFromKV -DomainShortName $DomainShortName -Workload $WorkloadName

$ExportParameters = @{
    Path              = $env:TEMP
    ConfigurationName = "ConfigureMicrosoft365"
    TenantId          = $TenantId
    Validate          = [Switch]$true
}
$ExportParameters.FileName = "M365Configuration-{0}.ps1" -f $WorkloadName
$ExportParameters.Workloads = $WorkloadName
$ExportParameters.ApplicationId = $ApplicationId
$ExportParameters.CertificateThumbprint = $CertThumbprint
try {
    Export-M365DSCConfiguration @ExportParameters
}
catch {
    throw $_
}
DennisLangenberg commented 1 year ago

Temporary workaround for me was to uninstall graph as a whole and let the correct version be installed by the DSC module.

remove conflicting graph modules first

Uninstall-Module -Name Microsoft.Graph -Force Get-InstalledModule Microsoft.Graph.* | ForEach-Object { if ($.Name -ne "Microsoft.Graph.Authentication") { Uninstall-Module $.Name } } Uninstall-Module Microsoft.Graph.Authentication

Install-Module Microsoft365DSC -requiredversion $verifiedversion -Force -AllowClobber # installs the module for desired state configuration management' Get-Module Microsoft365DSC -ListAvailable | select Name, Version Update-M365DSCDependencies -Force Import-Module Microsoft365DSC -Force

ricmestre commented 1 year ago

@DennisLangenberg That won't solve the issue I have now since it's a different thing.

Anyway that was one of the first workarounds I tried to solve my initial issue and it didn't work.

DennisLangenberg commented 1 year ago

@DennisLangenberg That won't solve the issue I have now since it's a different thing.

Anyway that was one of the first workarounds I tried to solve my initial issue and it didn't work.

Sorry to hear that for me this resolved the issue when checking blueprints and for now it is working (version M365DSC '1.23.628.1' was used)

ricmestre commented 1 year ago

@DennisLangenberg That's the thing, I'm using that DSC version exactly because it's the last one I've working that uses Graph v1.0, but the issue I've here is with later version which use Graph v2.0.

catalinpopa247 commented 1 year ago

Is there any progress on this @andikrueger? I followed the conversation so far but it seems far from resolved.

So, just to get it straight: I'm not using DSC, but a normal PowerShell script that I'm executing in a release pipeline in Azure DevOps, on a public agent. Any PowerShell script that I have and it's using Graph commands ends up in this error:

[error]Method not found: 'Void Microsoft.Graph.Authentication.AzureIdentityAccessTokenProvider..ctor(Azure.Core.TokenCredential, System.String[], Microsoft.Kiota.Authentication.Azure.ObservabilityOptions, System.String[])'.

image

I'm not forcing any version of Microsoft.Graph module, it takes the "default" official one from the "Azure Powershell" job available in Azure DevOps release pipelines. Also, the job for Azure PowerShell is set to use the latest PowerShell version available:

image

Just to give you an idea - the script is pretty simple, but the pipeline/job fails even at Get-MgUser command with that error:

image

ricmestre commented 1 year ago

@catalinpopa247 If your issue is not with DSC please refer to link [0] where what you're complaining was reported, in any case here's a few things that can help you get going, you only need one of them.

1) Set AzurePowerShell@5 with azurePowerShellVersion = 9.2.0

2) Set AzurePowerShell@5 with pwsh = true

3) Update to latest Graph v2.1 before running your commands (you get v2.0 installed by default), if you used regular PowerShell@2 task you'd have to call Connect-AzAccount first before importing Graph modules otherwise you'd get other problems.

[0] https://github.com/microsoftgraph/msgraph-sdk-powershell/issues/2148

catalinpopa247 commented 1 year ago

@ricmestre I would complain to that link, but I see also a lot of non-relevant talks in there about the real issue. The real issue here is that most probably somebody did an update on the public agents, along with the Graph v2.0 module and that didn't go well - and that resulted in literally having any PS script with Graph not working on public DevOps agents.

While I appreciate the workaround, I expect Microsoft to actually correct the bug they have created. That's why I am asking if there is any real update on this, apart from the workarounds that we all tried to figure out. I'm honestly more inclined now to temporarily execute things locally (where any configuration of Graph module v2.0 with any other tool/SDK works), rather than literally implementing a workaround on 50+ powershell scripts just because somebody from MS pushed an update on the public DevOps agents and on the V2.0 Graph module and didn't test before properly.