search

Azure / azure-powershell

Microsoft Azure PowerShell
Other
4.27k stars 3.87k forks source link

Azure Bicep Deployment Issues #26752

Open oZakari opened 6 days ago

oZakari commented 6 days ago

Description

Bug Report: Azure PowerShell Version 13.0.0 Causes Deployment Failures in ALZ-Bicep

Description:
Using Azure PowerShell version 13.0.0 has led to multiple reports of ALZ-Bicep deployment failures in both Azure DevOps pipelines and GitHub Actions workflows, where the latest version is specified. The issue is also reproducible in local environments.

The problem appears to impact specific modules within the ALZ-Bicep framework. In particular, I have confirmed failures when deploying the ALZ Default Policy Assignments module.

The following error is encountered during deployment:
Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject.

Note: For the failing deployment, we do load in a json file using the Bicep function: loadJsonContent

Workaround:
Pinning Azure PowerShell to version 12.5.0 or 12.4.0 resolves the issue.

Impact:
This issue affects deployments in:

Initial Reports of Error Here: https://github.com/Azure/ALZ-Bicep/issues/907

Issue script & Debug output

New-AzManagementGroupDeployment @inputObject
VERBOSE: Using Bicep v0.31.92
VERBOSE: Calling Bicep with arguments: build "C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\alzDefaults\alzDefaultPolicyAssignments.bicep" --stdout
WARNING: C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\CRML\customerUsageAttribution\cuaIdManagementGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentManagementGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\policyAssignmentManagementGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\alzDefaults\alzDefaultPolicyAssignments.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\assignments\alzDefaults\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentResourceGroup.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\policy\exemptions\policyExemptions.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentResourceGroupMany.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentManagementGroupMany.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentSubscriptionMany.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\modules\roleAssignments\roleAssignmentSubscription.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).
C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\CRML\customerUsageAttribution\cuaIdSubscription.bicep(1,1) : Info Bicep Linter Configuration: Custom bicepconfig.json file found (C:\Repos\ALZ\ALZ-Bicep\infra-as-code\bicep\bicepconfig.json).

WARNING: The DeploymentDebug setting has been enabled. This can potentially log secrets like passwords used in resource property or listKeys operations when you retrieve the deployment operations through Get-AzResourceGroupDeploymentOperation
New-AzManagementGroupDeployment: 
Line |
  11 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 7:34:37 PM - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject
New-AzManagementGroupDeployment: 
Line |
  11 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The deployment validation failed

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.6
PSEdition                      Core
GitCommitId                    7.4.6
OS                             Microsoft Windows 10.0.26100
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

Version    Name            Repository           Description
-------    ----                                ----------
13.0.0     Az              PSGallery            Microsoft Azure PowerShell - Cmdlets to …

Error output

Message        : The deployment validation failed
StackTrace     :    at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.HandleException(ExceptionDispatchInfo capturedException)     
                    at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : System.InvalidOperationException
InvocationInfo : {New-AzManagementGroupDeployment}
Line           : New-AzManagementGroupDeployment @inputObject
Position       : At line:9 char:1
                 + New-AzManagementGroupDeployment @inputObject
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 8

Message        : 2:35:29 PM - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject

StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzManagementGroupDeployment}
Line           : New-AzManagementGroupDeployment @inputObject
Position       : At line:9 char:1
                 + New-AzManagementGroupDeployment @inputObject
                 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 8
msJinLei commented 4 days ago

@oZakari Please set

$DebugPreference='Continue'

before you run the cmdlet and collect the debug

oZakari commented 3 days ago

Hi @msJinLei, here you go. Above this are just bunch of other similar types of warnings for the policy deployments

{
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-sandbox/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-zhov76caaytik",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-eh6uinipiyc62",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-vasi3f2dclqii",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-o3ippdiysur7o",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-p7acceslthje6",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      },
      {
        "level": "Warning",
        "code": "NestedDeploymentSkippedFromInternalExpansion",
        "target": "/providers/Microsoft.Management/managementGroups/alz-landingzones-corp/providers/Microsoft.Resources/deployments/pid-78001e36-9738-429c-a343-45cc84e8a527-w7ujuawx46ppo",
        "message": "When nested deployments are expanded, all its inner resources are retrieved for further validation. This process is performed in batch of: '10' at a time. Nested deployments exceeding this batch count are skipped from expansion."
      }
    ]
  }
}

DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [EnableErrorRecordsPersistence], Module = [], Cmdlet = []. Returning default value [False].
New-AzManagementGroupDeployment: 
Line |
  10 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | 8:58:31 AM - Error: Code=; Message=Received unexpected type Newtonsoft.Json.Linq.JObject
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - [ResourceManagerCmdletBase.ExecuteCmdlet] Caught unhandled exception: System.InvalidOperationException: The deployment validation failed
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkClient.NewResourceManagerSdkClient.RunDeploymentValidation(PSDeploymentCmdletParameters parameters, Deployment deployment)
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkClient.NewResourceManagerSdkClient.ExecuteDeploymentInternal(PSDeploymentCmdletParameters parameters)
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.SdkClient.NewResourceManagerSdkClient.ExecuteDeployment(PSDeploymentCmdletParameters parameters)
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.CmdletBase.DeploymentCreateCmdlet.ExecuteDeployment()
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.CmdletBase.DeploymentCreateCmdlet.OnProcessRecord()
   at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.ExecuteCmdlet()
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [EnableErrorRecordsPersistence], Module = [], Cmdlet = []. Returning default value [False].
New-AzManagementGroupDeployment: 
Line |
  10 |  New-AzManagementGroupDeployment @inputObject
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The deployment validation failed
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [DisplayRegionIdentified], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [CheckForUpgrade], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: AzureQoSEvent:  Module: Az.Resources:7.7.0; CommandName: New-AzManagementGroupDeployment; PSVersion: 7.4.6; IsSuccess: False; Duration: 00:00:25.5317021; SanitizeDuration: 00:00:00; Exception: The deployment validation failed;
DEBUG: 8:58:32 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 8:58:32 AM - NewAzureManagementGroupDeploymentCmdlet end processing.
DEBUG: Setting WindowTitle: Admin: ALZ-Bicep [update-known-issues] - PowerShell 7.4 (34216)
alex-frankel commented 2 days ago

@msJinLei - based on the discussion in this thread, it looks like the issue was introduced in Azure PowerShell 13. Is it possible this was introduced due to a change in some of the core components in Azure PowerShell? I am not sure that we have made many (or any) changes to the New-Az*Deployment commands recently.

cc @jeskew / @anthony-c-martin

anthony-c-martin commented 2 days ago

@tsmallig33 - I think this is related to #26732.

Looks like the error being returned in the error ("Received unexpected type Newtonsoft.Json.Linq.JObject") is coming from this code path, though it's not clear why: https://github.com/Azure/azure-powershell/blob/1bbb74dc2d5a883cc4be0a872d409c290621ec2e/src/Resources/ResourceManager/SdkClient/NewResourceManagerSdkClient.cs#L498

msJinLei commented 1 day ago

@tsmallig33 - I think this is related to #26732.

Looks like the error being returned in the error ("Received unexpected type Newtonsoft.Json.Linq.JObject") is coming from this code path, though it's not clear why:

azure-powershell/src/Resources/ResourceManager/SdkClient/NewResourceManagerSdkClient.cs

Line 498 in 1bbb74d

throw new InvalidOperationException($"Received unexpected type {validationResult.GetType()}");

I believe the PR is not released yet. Please correct me if anything incorrect @VeryEarly

tsmallig33 commented 1 day ago

@tsmallig33 - I think this is related to #26732. Looks like the error being returned in the error ("Received unexpected type Newtonsoft.Json.Linq.JObject") is coming from this code path, though it's not clear why: azure-powershell/src/Resources/ResourceManager/SdkClient/NewResourceManagerSdkClient.cs Line 498 in 1bbb74d throw new InvalidOperationException($"Received unexpected type {validationResult.GetType()}");

I believe the PR is not released yet. Please correct me if anything incorrect @VeryEarly

I think this was the PR where the deployments SDK was updated which looks to be related: https://github.com/Azure/azure-powershell/pull/26507

msJinLei commented 1 day ago

@tsmallig33 I think this was the PR where the deployments SDK was updated which looks to be related: #26507

We don't upgrade newtonsoft json library recently. Could you take a look?