New-AzResourceGroupDeployment got stuck with bicep templates

HeKr09 commented 10 months ago

Description

I am working with bicep templates to deploy some Azure resources. I am using bicep 0.23.1.45101 (install by MSI package if important) and the Az PowerShell module 10.4.1. Everything worked fine. A few weeks ago I wanted to update to the Az version 11.0.0. After the update I tried to ran New-AzResourceGroupDeployment. The cmdlet got stuck without any error. I needed to close the session. I had to go back to 10.4.1. Now I wanted to update to 11.1.0. Same problem.

Can anybody help?

Issue script & Debug output

New-AzResourceGroupDeployment -TemplateFile $resourceTemplate.FullName -TemplateParameterFile "parameters.$CustomerName.json" -ResourceGroupName $resourceGroup.ResourceGroupName

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.9
PSEdition                      Core
GitCommitId                    7.3.9
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                PSEdition ExportedCommands
---------- -------    ---------- ----                                --------- ----------------
Script     11.1.0                Az                                  Core,Desk
Script     10.4.1                Az                                  Core,Desk
Script     2.13.2                Az.Accounts                         Core,Desk {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Ena…
Script     2.0.0                 Az.Advisor                          Core,Desk {Disable-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Get-AzAdvisorCon…
Script     6.0.0                 Az.Aks                              Core,Desk {Get-AzAksCluster, New-AzAksCluster, Remove-AzAksCluster, Import-AzAksCredential…}
Script     5.5.1                 Az.Aks                              Core,Desk {Get-AzAksCluster, New-AzAksCluster, Remove-AzAksCluster, Import-AzAksCredential…}
Script     1.1.4                 Az.AnalysisServices                 Core,Desk {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysis…
Script     4.0.2                 Az.ApiManagement                    Core,Desk {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToProduct, Add-AzApiManage…
Script     1.0.0                 Az.App                              Core,Desk {Disable-AzContainerAppRevision, Enable-AzContainerAppRevision, Get-AzContainerApp…
Script     1.3.0                 Az.AppConfiguration                 Core,Desk {Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKey, New-AzAppConfigurati…
Script     2.2.2                 Az.ApplicationInsights              Core,Desk {Get-AzApplicationInsights, Get-AzApplicationInsightsApiKey, Get-AzApplicationInsi…
Script     1.0.0                 Az.ArcResourceBridge                Core,Desk {Get-AzArcResourceBridge, Get-AzArcResourceBridgeApplianceCredential, Get-AzArcRes…
Script     2.0.0                 Az.Attestation                      Core,Desk {Get-AzAttestationPolicy, Set-AzAttestationPolicy, Reset-AzAttestationPolicy, Get-…
Script     1.0.0                 Az.Automanage                       Core,Desk {Get-AzAutomanageBestPractice, Get-AzAutomanageConfigProfile, Get-AzAutomanageConf…
Script     1.9.1                 Az.Automation                       Core,Desk {Set-AzAutomationPython3Package, New-AzAutomationPython3Package, Remove-AzAutomati…
Script     3.5.0                 Az.Batch                            Core,Desk {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBatchAcco…
Script     2.0.3                 Az.Billing                          Core,Desk {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzConsump…
Script     3.1.1                 Az.Cdn                              Core,Desk {Clear-AzCdnEndpointContent, Clear-AzFrontDoorCdnEndpointContent, Disable-AzCdnCus…
Script     2.0.0                 Az.CloudService                     Core,Desk {Get-AzCloudService, Get-AzCloudServiceInstanceView, Get-AzCloudServiceNetworkInte…
Script     1.2.0                 Az.CloudService                     Core,Desk {Get-AzCloudService, Get-AzCloudServiceInstanceView, Get-AzCloudServiceNetworkInte…
Script     1.14.1                Az.CognitiveServices                Core,Desk {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiv…
Script     1.14.0                Az.CognitiveServices                Core,Desk {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiv…
Script     7.1.0                 Az.Compute                          Core,Desk {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-Az…
Script     6.3.0                 Az.Compute                          Core,Desk {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-Az…
Script     1.0.0                 Az.ConfidentialLedger               Core,Desk {Get-AzConfidentialLedger, New-AzConfidentialLedger, New-AzConfidentialLedgerAADBa…
Script     4.0.0                 Az.ContainerInstance                Core,Desk {Add-AzContainerInstanceOutput, Get-AzContainerGroup, Get-AzContainerInstanceCache…
Script     3.2.2                 Az.ContainerInstance                Core,Desk {Add-AzContainerInstanceOutput, Get-AzContainerGroup, Get-AzContainerInstanceCache…
Script     4.1.2                 Az.ContainerRegistry                Core,Desk {Connect-AzContainerRegistry, Get-AzContainerRegistryRepository, Remove-AzContaine…
Script     4.1.1                 Az.ContainerRegistry                Core,Desk {Connect-AzContainerRegistry, Get-AzContainerRegistryRepository, Remove-AzContaine…
Script     1.13.0                Az.CosmosDB                         Core,Desk {Get-AzCosmosDBSqlContainer, Get-AzCosmosDBSqlContainerThroughput, Get-AzCosmosDBS…
Script     1.12.0                Az.CosmosDB                         Core,Desk {Get-AzCosmosDBSqlContainer, Get-AzCosmosDBSqlContainerThroughput, Get-AzCosmosDBS…
Script     1.1.0                 Az.DataBoxEdge                      Core,Desk {Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeDevice, Invoke-AzDataBoxEdgeDevice, New-Az…
Script     1.7.1                 Az.Databricks                       Core,Desk {Get-AzDatabricksAccessConnector, Get-AzDatabricksOutboundNetworkDependenciesEndpo…
Script     1.7.0                 Az.Databricks                       Core,Desk {Get-AzDatabricksAccessConnector, Get-AzDatabricksOutboundNetworkDependenciesEndpo…
Script     1.18.0                Az.DataFactory                      Core,Desk {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzDataFa…
Script     1.17.0                Az.DataFactory                      Core,Desk {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzDataFa…
Script     1.0.3                 Az.DataLakeAnalytics                Core,Desk {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Remo…
Script     1.3.0                 Az.DataLakeStore                    Core,Desk {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, Re…
Script     2.1.0                 Az.DataProtection                   Core,Desk {Backup-AzDataProtectionBackupInstanceAdhoc, Edit-AzDataProtectionPolicyRetentionR…
Script     1.0.1                 Az.DataShare                        Core,Desk {New-AzDataShareAccount, Get-AzDataShareAccount, Remove-AzDataShareAccount, New-Az…
Script     1.1.0                 Az.DeploymentManager                Core,Desk {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentManagerArtifactSource, Set…
Script     4.2.0                 Az.DesktopVirtualization            Core,Desk {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdAppAttachPackage, Ge…
Script     4.0.0                 Az.DesktopVirtualization            Core,Desk {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdApplication, Get-AzW…
Script     1.0.0                 Az.DevCenter                        Core,Desk {Get-AzDevCenterAdminAttachedNetwork, Get-AzDevCenterAdminCatalog, Get-AzDevCenter…
Script     1.0.2                 Az.DevTestLabs                      Core,Desk {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPol…
Script     1.2.0                 Az.Dns                              Core,Desk {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecord…
Script     1.1.3                 Az.Dns                              Core,Desk {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecord…
Script     1.6.0                 Az.EventGrid                        Core,Desk {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGrid…
Script     4.2.0                 Az.EventHub                         Core,Desk {New-AzEventHubAuthorizationRuleSASToken, Approve-AzEventHubPrivateEndpointConnect…
Script     4.1.0                 Az.EventHub                         Core,Desk {New-AzEventHubAuthorizationRuleSASToken, Approve-AzEventHubPrivateEndpointConnect…
Script     1.10.0                Az.FrontDoor                        Core,Desk {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-AzFrontDoor…}
Script     4.0.7                 Az.Functions                        Core,Desk {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get…
Script     4.0.6                 Az.Functions                        Core,Desk {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get…
Script     6.0.2                 Az.HDInsight                        Core,Desk {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New-A…
Script     6.0.1                 Az.HDInsight                        Core,Desk {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New-A…
Script     2.0.0                 Az.HealthcareApis                   Core,Desk {Get-AzHealthcareApisService, Get-AzHealthcareApisWorkspace, Get-AzHealthcareDicom…
Script     2.7.5                 Az.IotHub                           Core,Desk {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionString,…
Script     5.0.1                 Az.KeyVault                         Core,Desk {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertifica…
Script     4.12.0                Az.KeyVault                         Core,Desk {Add-AzKeyVaultCertificate, Update-AzKeyVaultCertificate, Stop-AzKeyVaultCertifica…
Script     2.3.0                 Az.Kusto                            Core,Desk {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAtt…
Script     2.2.0                 Az.Kusto                            Core,Desk {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAtt…
Script     1.0.0                 Az.LoadTesting                      Core,Desk {Get-AzLoad, New-AzLoad, Remove-AzLoad, Update-AzLoad}
Script     1.5.0                 Az.LogicApp                         Core,Desk {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegr…
Script     1.1.3                 Az.MachineLearning                  Core,Desk {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitment…
Script     1.0.0                 Az.MachineLearningServices          Core,Desk {Get-AzMLServiceQuota, Get-AzMLServiceUsage, Get-AzMLServiceVMSize, Get-AzMLWorksp…
Script     1.4.1                 Az.Maintenance                      Core,Desk {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration,…
Script     1.3.1                 Az.Maintenance                      Core,Desk {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration,…
Script     1.2.0                 Az.ManagedServiceIdentity           Core,Desk {Get-AzSystemAssignedIdentity, Get-AzUserAssignedIdentity, Get-AzUserAssignedIdent…
Script     1.1.1                 Az.ManagedServiceIdentity           Core,Desk {Get-AzFederatedIdentityCredentials, Get-AzSystemAssignedIdentity, Get-AzUserAssig…
Script     3.0.0                 Az.ManagedServices                  Core,Desk {Get-AzManagedServicesAssignment, Get-AzManagedServicesDefinition, Get-AzManagedSe…
Script     2.0.0                 Az.MarketplaceOrdering              Core,Desk {Get-AzMarketplaceTerms, Invoke-AzMarketplaceSignTerms, Set-AzMarketplaceTerms, St…
Script     1.1.2                 Az.Media                            Core,Desk {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-AzMediaServiceKey, Get-…
Script     2.2.0                 Az.Migrate                          Core,Desk {Get-AzMigrateDiscoveredServer, Get-AzMigrateJob, Get-AzMigrateProject, Get-AzMigr…
Script     5.0.0                 Az.Monitor                          Core,Desk {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile…}
Script     4.6.0                 Az.Monitor                          Core,Desk {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile…}
Script     1.1.1                 Az.MySql                            Core,Desk {Get-AzMySqlConfiguration, Get-AzMySqlConnectionString, Get-AzMySqlFirewallRule, G…
Script     7.1.0                 Az.Network                          Core,Desk {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthen…
Script     6.2.0                 Az.Network                          Core,Desk {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthen…
Script     1.0.0                 Az.NetworkCloud                     Core,Desk {Deploy-AzNetworkCloudCluster, Disable-AzNetworkCloudStorageApplianceRemoteVendorM…
Script     1.1.2                 Az.NotificationHubs                 Core,Desk {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotification…
Script     3.2.0                 Az.OperationalInsights              Core,Desk {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsCus…
Script     1.6.4                 Az.PolicyInsights                   Core,Desk {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemed…
Script     1.6.3                 Az.PolicyInsights                   Core,Desk {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemed…
Script     1.1.0                 Az.PostgreSql                       Core,Desk {Get-AzPostgreSqlConfiguration, Get-AzPostgreSqlConnectionString, Get-AzPostgreSql…
Script     2.0.0                 Az.PowerBIEmbedded                  Core,Desk {Resume-AzPowerBIEmbeddedCapacity, Suspend-AzPowerBIEmbeddedCapacity, Get-AzPowerB…
Script     1.2.1                 Az.PowerBIEmbedded                  Core,Desk {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPowe…
Script     1.0.4                 Az.PrivateDns                       Core,Desk {Get-AzPrivateDnsZone, Remove-AzPrivateDnsZone, Set-AzPrivateDnsZone, New-AzPrivat…
Script     6.6.2                 Az.RecoveryServices                 Core,Desk {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecovery…
Script     6.6.0                 Az.RecoveryServices                 Core,Desk {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecovery…
Script     1.8.1                 Az.RedisCache                       Core,Desk {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCache…
Script     1.8.0                 Az.RedisCache                       Core,Desk {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCache…
Script     1.2.0                 Az.RedisEnterpriseCache             Core,Desk {Export-AzRedisEnterpriseCache, Get-AzRedisEnterpriseCache, Get-AzRedisEnterpriseC…
Script     2.0.0                 Az.Relay                            Core,Desk {Get-AzRelayAuthorizationRule, Get-AzRelayHybridConnection, Get-AzRelayKey, Get-Az…
Script     1.2.0                 Az.ResourceMover                    Core,Desk {Add-AzResourceMoverMoveResource, Get-AzResourceMoverMoveCollection, Get-AzResourc…
Script     6.12.1                Az.Resources                        Core,Desk {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRol…
Script     6.11.1                Az.Resources                        Core,Desk {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRol…
Script     1.5.1                 Az.Security                         Core,Desk {Get-AzSecuritySolution, Get-AzSecuritySolutionsReferenceData, New-AzAlertsSuppres…
Script     1.4.0                 Az.Security                         Core,Desk {Get-AzSecuritySolution, Get-AzSecuritySolutionsReferenceData, New-AzAlertsSuppres…
Script     3.1.1                 Az.SecurityInsights                 Core,Desk {Get-AzSentinelAlertRule, Get-AzSentinelAlertRuleAction, Get-AzSentinelAlertRuleTe…
Script     3.1.0                 Az.SecurityInsights                 Core,Desk {Get-AzSentinelAlertRule, Get-AzSentinelAlertRuleAction, Get-AzSentinelAlertRuleTe…
Script     3.0.0                 Az.ServiceBus                       Core,Desk {New-AzServiceBusAuthorizationRuleSASToken, Test-AzServiceBusNameAvailability, App…
Script     3.3.1                 Az.ServiceFabric                    Core,Desk {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricNode, Add-AzServiceFabri…
Script     3.2.0                 Az.ServiceFabric                    Core,Desk {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricNode, Add-AzServiceFabri…
Script     2.0.0                 Az.SignalR                          Core,Desk {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey…}
Script     4.12.0                Az.Sql                              Core,Desk {Invoke-AzSqlServerExternalGovernanceStatusRefresh, Get-AzSqlDatabaseTransparentDa…
Script     4.10.0                Az.Sql                              Core,Desk {Invoke-AzSqlServerExternalGovernanceStatusRefresh, Get-AzSqlDatabaseTransparentDa…
Script     2.1.0                 Az.SqlVirtualMachine                Core,Desk {Get-AzAvailabilityGroupListener, Get-AzSqlVM, Get-AzSqlVMGroup, Invoke-AzRedeploy…
Script     2.2.3                 Az.StackHCI                         Core,Desk {Add-AzStackHCIVMAttestation, Disable-AzStackHCIAttestation, Disable-AzStackHCIRem…
Script     2.2.0                 Az.StackHCI                         Core,Desk {Add-AzStackHCIVMAttestation, Disable-AzStackHCIAttestation, Disable-AzStackHCIRem…
Script     6.0.1                 Az.Storage                          Core,Desk {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorag…
Script     5.10.1                Az.Storage                          Core,Desk {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorag…
Script     1.2.0                 Az.StorageMover                     Core,Desk {Get-AzStorageMover, Get-AzStorageMoverAgent, Get-AzStorageMoverEndpoint, Get-AzSt…
Script     1.0.1                 Az.StorageMover                     Core,Desk {Get-AzStorageMover, Get-AzStorageMoverAgent, Get-AzStorageMoverEndpoint, Get-AzSt…
Script     2.1.0                 Az.StorageSync                      Core,Desk {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzStorageSy…
Script     2.0.0                 Az.StorageSync                      Core,Desk {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzStorageSy…
Script     2.0.0                 Az.StreamAnalytics                  Core,Desk {Get-AzStreamAnalyticsCluster, Get-AzStreamAnalyticsClusterStreamingJob, Get-AzStr…
Script     1.0.0                 Az.Support                          Core,Desk {Get-AzSupportService, Get-AzSupportProblemClassification, Get-AzSupportTicket, Ge…
Script     3.0.4                 Az.Synapse                          Core,Desk {Get-AzSynapseSparkJob, Stop-AzSynapseSparkJob, Submit-AzSynapseSparkJob, Wait-AzS…
Script     3.0.3                 Az.Synapse                          Core,Desk {Get-AzSynapseSparkJob, Stop-AzSynapseSparkJob, Submit-AzSynapseSparkJob, Wait-AzS…
Script     1.2.2                 Az.TrafficManager                   Core,Desk {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFr…
Script     1.2.1                 Az.TrafficManager                   Core,Desk {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFr…
Script     3.1.2                 Az.Websites                         Core,Desk {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppSer…
Script     3.1.1                 Az.Websites                         Core,Desk {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppSer…

Error output

None

HeKr09 commented 10 months ago

Anyone an idea?

anthony-c-martin commented 10 months ago

@HeKr09 could you try the following and share the output?

Enable detailed Bicep tracing:
```
$env:BICEP_TRACING_ENABLED="true"
```

Re-run your command with the -Verbose flag:

New-AzResourceGroupDeployment -TemplateFile $resourceTemplate.FullName -TemplateParameterFile "parameters.$CustomerName.json" -ResourceGroupName $resourceGroup.ResourceGroupName -Verbose

lansalot commented 10 months ago

Yes, same here, as referenced at https://github.com/Azure/bicep/issues/10350#issuecomment-1831547335

I found changing vscode to use powershell5.1 instead of pscore got it working again. You'll notice a hung bicep.exe process, that's what's stalling the deployment?

anthony-c-martin commented 10 months ago

@lansalot it would be interesting to see if there's any difference in using .bicepparam files - the Bicep CLI is invoked in a slightly different manner (using stdout, rather than writing a temporary file).

If you're able to:

Create a main.bicepparam file with contents:
```
using 'main.bicep'

param foo = 'foo'
```

Create a main.bicep file with contents:

param foo string

output foo string = foo

Enable tracing and call Bicep:

$env:BICEP_TRACING_ENABLED = "true"
$rgName = "" #replace this with an RG name
New-AzResourceGroupDeployment -TemplateParameterFile "main.bicepparam" -ResourceGroupName $rgName -Verbose

HeKr09 commented 10 months ago

@anthony-c-martin I followed your steps. Same output than before:

VERBOSE: Using Bicep v0.23.1
VERBOSE: Calling Bicep with arguments: build "PATHTOBICEP" --outdir "C:\Users\my.user\AppData\Local\Temp\281977cc-4340-4668-8e43-4d1f45d920c9"

Got stuck. I waited five minutes...

HeKr09 commented 10 months ago

@anthony-c-martin Same with .bicepparam.

HeKr09 commented 10 months ago

I just wonder about the following. I run the following within my PowerShell:

Import-Module -Name Az -RequiredVersion 11.1.0 -Force
New-AzResourceGroupDeployment -ResourceGroupName "RG" -TemplateParameterFile "TemplateParam.bicepparam" -TemplateFile "Template.bicep" -Verbose

I just wonder about the fact that I am typing the Az cmdlet and when I have got this line printed:

New-AzResourceGroupDeployment -ResourceGroupName "RG" -TemplateParameterFile "TemplateParam.bicepparam"

When I then press TAB to see the other parameters my whole PowerShell got stuck. I can't click anything...

HeKr09 commented 9 months ago

Anyone an idea?

anthony-c-martin commented 9 months ago

One thing that's especially odd here - I can tell from your logs that the cmdlet was able to invoke Bicep successfully the first time (VERBOSE: Using Bicep v0.23.1) to obtain the version number, so this isn't a generic issue with the mechanism we use to invoke bicep.exe.

I can't think of anything significantly different in the way the cmdlet invokes Bicep the 2nd time (to perform the build), so something like WDAC blocking certain operations (e.g. file I/O) feels like a plausible explanation. Unfortunately this is not my area of expertise. If you have the time, it might be worth trying to follow this guide to see if there's anything to indicate a WDAC issue.

@HeKr09 https://github.com/PowerShell/PowerShell/issues/20785 feels possibly related. Could you try and repro this after running the following in the same powershell window:

$ExecutionContext.SessionState.LanguageMode = 'FullLanguage'

HeKr09 commented 9 months ago

@anthony-c-martin I followed the guide and could not detect any errors. I also set the LanguageMode and got the same output and behavior than before.

Currently I am using PowerShell 7.3.10. According to your #20785 the problem occurs with 7.4.0 and the user should roll back to 7.3.10. So this does not to seem the problem. Also with Az 10.4.1 everything works fine, :(

vladimirs-davidovs commented 9 months ago

Hello everyone! I am having the same problem with:

Bicep CLI v0.24.24
PowerShell v7.4.0
Latest AZ PS modules

What I also noticed was that:

The issue does not seem to affect Bicep templates that are smaller than a certain size, e.g., if it only deploys an RG and a vNet, deployment will commence, if it has a few referenced modules - it would 'hang'. I have not yet looked into correlation of particular template 'sizes' with the deployments 'hanging'.
The problem affects actual as well as -WhatIf deployments.
If deployment is bound to fail, then TAB autocompletion for New-Az****Deployment commands (it does not affect RG deployments only) stops working and shell simply 'hangs', in cases, when -TemplateFile parameter is set in the command line already.
If I run Sysinternals DbgView before attempting the deployment, I can see Bicep runtime reporting processing of .bicep template files via debug logging then at the time, when the command line freezes, logging stops until I kill the Bicep process, after which Bicep runtime processes more .bicep templates and then debug logging stops.
Deployment works fine from the PowerShell running in WSL.

Has anyone found a solution to that?

OlehGap commented 9 months ago

the same here.

New-AzResourceGroupDeployment and New-AzSubscriptionDeployment have suddenly stopped working in the mid of the day. Last verbose messages

VERBOSE: Using Bicep v0.24.24
VERBOSE: Calling Bicep with arguments: build ...

However, if i run bicep with the same params outside of VSCode, it works fine.

I've noticed from Sysinternals Process Explorer process tree last process conhost.exe is hanging code.exe -- pwsh.exe ---- bicep.exe ------ conhost.exe (\??\C:\WINDOWS\system32\conhost.exe 0x4)

anthony-c-martin commented 9 months ago

@OlehGap that's very helpful. Is it possible to see the cli arguments that conhost was invoked with?

The only thing that comes to mind for me is that the sdk we use for Azure auth invokes AzCLI or AzPwsh via shell to obtain an auth token.

Are you using template spec or module registry modules anywhere in your deployment?

OlehGap commented 9 months ago

@anthony-c-martin, nothing fancy here, just developing sub modules

` $VerbosePreference = 'Continue'

$InformationPreference = 'Continue'

New-AzResourceGroupDeployment -ResourceGroupName $rgName -TemplateFile .\Modules\resourceGroup-RoleAssignments.bicep -DeploymentDebugLogLevel All -Pre -roleDefinitionIdOrName 'Custom1' -principalId $principalId `

I guess, I've found the issue - bicep gets stuck on big files. The deployment worked until I copied _builtInRoleNamesvar from CARML/AVM/BRM repos to include all Azure built-in roles into my module.

from here https://github.com/Azure/bicep-registry-modules/blob/main/modules/lz/sub-vending/src/carml/v0.6.0/Microsoft.Authorization/roleAssignments/resourceGroup/deploy.bicep

Once I reduced the number of roles, the deployment went back to normal.

Turns out bicep.exe cannot ingest those extra 35KB of data

anthony-c-martin commented 9 months ago

Once I reduced the number of roles, the deploy Turns out bicep.exe cannot ingest those extra 35KB of data

That's a helpful data point! If you invoke the Bicep build command directly with the same parameters that the AzPwsh command does, do you see the same behavior?

OlehGap commented 9 months ago

That's correct. New-AzResourceGroupDeployment - hangs on calling bicep ( with extra child process - see a screenshot in my first post) Executing bicep.exe directly with the same params works as a charm.

I presume the settings are different. bicepconfig.json or some file in .bicep folders ???

anthony-c-martin commented 9 months ago

The Bicep CLI command is executed exactly the same way via Azure PowerShell - there should be no difference in arguments or settings passed in.

~~I suspect we may be running into the deadlock condition described under https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.process.standardoutput?view=net-8.0&redirectedfrom=MSDN#remarks.~~

EDIT: Ruled this out - we actually run into this problem when not using stdout.

anthony-c-martin commented 9 months ago

@OlehGap if you get a chance, could you test out using a .bicepparam file without referencing the .bicep file in the cmdlet with the -TemplateFile parameter?

For example, something like:

New-AzResourceGroupDeployment -ResourceGroupName $rgName -TemplateParameterFile .\Modules\resourceGroup-RoleAssignments.bicepparam

Where your .bicepparam file could be:

using './resourceGroup-RoleAssignments.bicepparam'

// fill these out with your parameter values
param roleDefinitionIdOrName = ...
param principalId = ...

Building a .bicep file requires us to create a temporary directory and write to disk, whereas building a .bicepparam file doesn't have this dependency. This would allow us to rule in / out writing to disk as the cause.

HeKr09 commented 8 months ago

Does anyone has a solution yet?

anthony-c-martin commented 8 months ago

@HeKr09 if you're able to answer my previous comment, that would help us figure out the solution.

HeKr09 commented 8 months ago

@anthony-c-martin Half win: The PowerShell got stuck when using TAB to get the next parameter. Nevertheless the PowerShell ran after I wrote it myself: New-AzResourceGroupDeployment -ResourceGroupName "TEST_RG" -TemplateParameterFile '.\parameters.TEST.bicepparam' -Verbose

The output:

VERBOSE: Using Bicep v0.23.1
VERBOSE: Calling Bicep with arguments: build-params "D:\Customer Templates\Test\Bicep\parameters.Test.bicepparam" --stdout
VERBOSE: Performing the operation "Creating Deployment" on target "TEST_RG".
VERBOSE: Using Bicep v0.23.1
VERBOSE: Calling Bicep with arguments: build-params "D:\Customer Templates\Test\Bicep\parameters.Test.bicepparam" --stdout
WARNING: D:\Customer Templates\Test\Bicep\templateLogicApp.Test.bicep(25,8) : Warning explicit-values-for-loc-params: Parameter 'location' of module 'ModuleConnections' isn't assigned an explicit value, and its default value may not give the intended behavior for a location-related parameter. You should assign an explicit value to the parameter. [https://aka.ms/bicep/linter/explicit-values-for-loc-params]

VERBOSE: 06:11:28 - Template is valid.
VERBOSE: 06:11:29 - Create template deployment '3ad712ee-e079-4737-a87a-2513417423b0'
VERBOSE: 06:11:29 - Checking deployment status in 5 seconds
VERBOSE: 06:11:35 - Resource Microsoft.Resources/deployments 'connectionsDeploy' provisioning status is running
VERBOSE: 06:11:35 - Checking deployment status in 14 seconds
VERBOSE: 06:11:50 - Resource Microsoft.Resources/deployments 'connectionsDeploy' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Web/connections 'office365' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Web/connections 'keyvault' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Web/connections 'azuread' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Web/connections 'azurequeues' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Web/connections 'azurequeues-1' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Web/connections 'sharepointonline' provisioning status is succeeded
VERBOSE: 06:11:50 - Resource Microsoft.Resources/deployments 'connectionsDeploy' provisioning status is succeeded

DeploymentName          : 3ad712ee-e079-4737-a87a-2513417423b0
ResourceGroupName       : TEST_RG
ProvisioningState       : Succeeded
Timestamp               : 31.01.2024 05:11:47
Mode                    : Incremental
TemplateLink            :
Parameters              :
                          [SOMEHERE]

Outputs                 :
DeploymentDebugLogLevel :

But: WHY?!?!?! Can anybody explain? Is this a bug in the module?

anthony-c-martin commented 8 months ago

But: WHY?!?!?! Can anybody explain? Is this a bug in the module?

I'm still puzzled to be honest, but this gives me more confidence that #23997 may fix the problem.

The -ParameterFile argument without the -TemplateFile invokes the Bicep CLI, using the --stdout option, and reads the compiled template back via stdout.

The -TemplateFile argument invokes the Bicep CLI, but creates a temporary file on disk and then reads it back (rather than using --stdout). My change switches this cmdlet to instead use --stdout.

@wyunchi-ms is there a way to obtain a nightly/prerelease version of the cmdlet to test this out?

JoranSlingerland commented 8 months ago

Just checked with the latest az.resources 6.15.0. This seems to have the --stdout flag implemented in your pr. But it still hangs on the bicep build.

See my output below:

PS C:\Users\xxx\Azure_devops\azure-resources> import-module az.resources -requiredVersion 6.15.0
PS C:\Users\xxx\Azure_devops\azure-resources> new-azsubscriptionDeployment -resourcegroupname xxx-it-weu-xxx-2001 -location westeurope -TemplateFile .\
modules\Microsoft.Storage\storageAccounts\.test\common\deploy.test.bicep -Verbose
VERBOSE: Using Bicep v0.23.1
VERBOSE: Calling Bicep with arguments: build "C:\Users\xxx\Azure_devops\azure-resources\modules\Microsoft.Storage\storageAccounts\.test\common\deploy.test.
bicep" --stdout

anthony-c-martin commented 8 months ago

@JoranSlingerland thanks for sharing - at least we can rule out the "file writing" logic as the root cause I guess... 😞

So we're back to square one... Since I'm unable to reproduce this myself, I think the next step of investigation would be to try and capture a memory dump of the process:

Open Task manager
Go to the "Details" view
Wait for bicep.exe to be stuck for at least 60s
Right click on bicep.exe
Select "Create memory dump"
Share the file it creates (please note that this could contain sensitive information such as file paths and usernames - please email me at antmarti@microsoft.com rather than sharing it publicly on GitHub).

JoranSlingerland commented 8 months ago

@anthony-c-martin I've send you an email from my @achmea.nl work email with the memory dump.

I've also checked when the issues seems to be introduced and I've found that az.resources 6.11.1 still works but 6.11.2 doesn't work anymore.

anthony-c-martin commented 8 months ago

@JoranSlingerland thank you very much! Here's where the application appears to have hung:

Searching for the call stack doesn't turn up much; https://github.com/dotnet/runtime/issues/97496 seems possibly related.

In Bicep 0.25.3 (released yesterday) we switched from using .NET7 to .NET8 - could you try using this version of Bicep in case it makes a difference?

JoranSlingerland commented 8 months ago

@anthony-c-martin I've tried running on 0.25.3 but this still gives the same issue. I've send you another email with the dump file. See the powershell console output below:

PowerShell 7.4.1
PS C:\Users\xxx\Azure_devops\azure-resources> new-azsubscriptionDeployment -resourcegroupname xxx-it-weu-xxx-2001 -location westeurope -TemplateFile .\modules\Microsoft.Storage\storageAccounts\.test\common\deploy.test.bicep -Verbose
VERBOSE: Using Bicep v0.25.3
VERBOSE: Calling Bicep with arguments: build "C:\Users\xxx\Azure_devops\azure-resources\modules\Microsoft.Storage\storageAccounts\.test\common\deploy.test.bicep" --stdout

anthony-c-martin commented 8 months ago

@JoranSlingerland thank you! I think I've finally figured out what's going on - I've submitted a PR to fix it here: https://github.com/Azure/azure-powershell/pull/24133

The deadlock appears when a large number of errors or warning diagnostics are logged, so a temporary workaround would be to find ways of cutting down the number of diagnostics emitted - for example adding inline suppressions.

JoranSlingerland commented 8 months ago

@anthony-c-martin thank you for the update and fix! Will see if we can implement the workaround and otherwise wait for a new release.

anthony-c-martin commented 8 months ago

This has just been released under Az powershell version 11.3.1 - you should be able to update to pick up the fix: https://learn.microsoft.com/en-us/powershell/azure/install-azps-windows#update-the-az-powershell-module

Closing the issue, but please re-open or respond here if you're still having problems.

Azure / azure-powershell