[Question] Did the Behaviour of Restart-AzVM change from sync to async?

[Tornhoof]

Description

Recently one of my automated test scripts for a web application started failing constantly, after working fine for 2-3 years (I once migrated from the old AzureRM namespace). The script does the following steps:

1. Setup VM (Image Windows2019-datacenter-latest)
2. Add IIS, Enable WinRM
3. Restart AzVM (via Restart-AzVM -ResourceGroupName $resourceGroup -Name $vmName)
4. Run a few operations via WinRM, that takes like 20 minutes

As of a few weeks (maybe 2-3 weeks) ago this worked fine, and the Restart behaviour worked synchronously and it more or less immediately restarted the VM and busily waited until the restart was done and step 4 worked fine

As of now it apparently reboots in the middle of my Step 4 and the script aborts with connection loss errors. The Azure Portal Activity log for that machine lists a Restart VM entry at the correct time, but the actual Event log in the system with the appropriate system events about stopped services is like 5 minutes later.

I've now removed the Restart step from my script (it's not necessary anymore, maybe it was necessary back in the win2016 image times) and my scripts works again, so I currently conclude that some behaviour change happened to Restart-AzVM.

I currently have a few ideas what might be the culprit:

1. The behaviour of Restart-AzVM changed (probably on the server side, as it does not work with older Az Versions either)
2. The behaviour of the image changed, so it does not reboot immediately anymore but instead waits for some timeout (you know like Desktop and that "Open applications preventing reboot dialog").
3. My usage of Restart AzVM was bonkers and I should have never used it without some active polling to tell me when it actually restarted and it was sheer luck that it worked.

I kinda think it's something like 2 where e.g Windows Update running on the system is now preventing the reboot until it's done and previously was just working because nothing blocked.

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.19041.610
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.610
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

I also tried with the most recent 7.1, but no difference

Module versions

ModuleType Version    Name                                ExportedCommands
---------- -------    ----                                ----------------
Script     2.2.1      Az.Accounts                         {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave...}
Script     1.1.1      Az.Advisor                          {Get-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Disable-AzAdvisorRecommendation, Get-AzAdvisorConfiguration...}
Script     2.0.1      Az.Aks                              {Get-AzAksCluster, New-AzAksCluster, Remove-AzAksCluster, Import-AzAksCredential...}
Script     1.1.4      Az.AnalysisServices                 {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServicesServer, Remove-AzAnalysisServicesServer...}
Script     2.1.0      Az.ApiManagement                    {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiManagementRegion...}
Script     1.0.0      Az.AppConfiguration                 {Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKey, New-AzAppConfigurationStore, New-AzAppConfigurationStoreKey...}
Script     1.1.0      Az.ApplicationInsights              {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationInsights, Update-AzApplicationInsights...}
Script     1.4.0      Az.Automation                       {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Get-AzAutomationJobOutputRecord, Import-AzAutomationDscNodeConfiguration...}
Script     3.1.0      Az.Batch                            {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBatchAccount...}
Script     2.0.0      Az.Billing                          {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzConsumptionBudget...}
Script     1.6.0      Az.Cdn                              {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnProfile...}
Script     1.8.0      Az.CognitiveServices                {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiveServicesAccountSku, Get-AzCognitiveServicesAccountType...}
Script     4.6.0      Az.Compute                          {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-AzAvailabilitySet...}
Script     1.0.3      Az.ContainerInstance                {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-AzContainerInstanceLog}
Script     2.0.0      Az.ContainerRegistry                {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistry, Remove-AzContainerRegistry...}
Script     1.1.0      Az.DataBoxEdge                      {Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeDevice, Invoke-AzDataBoxEdgeDevice, New-AzDataBoxEdgeDevice...}
Script     1.0.1      Az.Databricks                       {Get-AzDatabricksVNetPeering, Get-AzDatabricksWorkspace, New-AzDatabricksVNetPeering, New-AzDatabricksWorkspace...}
Script     1.11.1     Az.DataFactory                      {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzDataFactoryV2...}
Script     1.0.2      Az.DataLakeAnalytics                {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Remove-AzDataLakeAnalyticsCatalogCredential, Set-AzDataLakeAnalyticsCatalogCredential...}
Script     1.3.0      Az.DataLakeStore                    {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreFirewallRule, Set-AzDataLakeStoreTrustedIdProvider...}
Script     1.0.0      Az.DataShare                        {New-AzDataShareAccount, Get-AzDataShareAccount, Remove-AzDataShareAccount, New-AzDataShare...}
Script     1.1.0      Az.DeploymentManager                {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentManagerArtifactSource, Set-AzDeploymentManagerArtifactSource, Remove-AzDeploymentManagerArtifactSource...}
Script     2.0.1      Az.DesktopVirtualization            {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdApplication, Get-AzWvdApplicationGroup...}
Script     1.0.2      Az.DevTestLabs                      {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy...}
Script     1.1.2      Az.Dns                              {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecordSet...}
Script     1.3.0      Az.EventGrid                        {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGridTopicKey...}
Script     1.7.1      Az.EventHub                         {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, Remove-AzEventHubNamespace...}
Script     1.6.1      Az.FrontDoor                        {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-AzFrontDoor...}
Script     2.0.0      Az.Functions                        {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get-AzFunctionAppSetting...}
Script     4.1.0      Az.HDInsight                        {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New-AzHDInsightStreamingMapReduceJobDefinition...}
Script     1.1.0      Az.HealthcareApis                   {New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthcareApisService, Get-AzHealthcareApisService}
Script     2.7.0      Az.IotHub                           {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionString, Get-AzIotHubJob...}
Script     3.1.0      Az.KeyVault                         {Add-AzManagedHsmKey, Get-AzManagedHsmKey, Remove-AzManagedHsmKey, Undo-AzManagedHsmKeyRemoval...}
Script     1.0.0      Az.Kusto                            {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAttachedDatabaseConfiguration, Get-AzKustoCluster...}
Script     1.4.0      Az.LogicApp                         {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationAccountBatchConfiguration, Get-AzIntegrationAccountCallbackUrl...}
Script     1.1.3      Az.MachineLearning                  {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlanUsageHistory, Remove-AzMlCommitmentPlan...}
Script     1.1.0      Az.Maintenance                      {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration, Get-AzMaintenanceUpdate...}
Script     2.0.0      Az.ManagedServices                  {Get-AzManagedServicesAssignment, New-AzManagedServicesAssignment, Remove-AzManagedServicesAssignment, Get-AzManagedServicesDefinition...}
Script     1.0.2      Az.MarketplaceOrdering              {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms}
Script     1.1.1      Az.Media                            {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-AzMediaServiceKey, Get-AzMediaServiceNameAvailability...}
Script     2.2.0      Az.Monitor                          {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile...}
Script     4.3.0      Az.Network                          {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthenticationCertificate, New-AzApplicationGatewayAuthenticationCertificate, Remove-AzApplicationGatewayAuthenticationCertificate...}
Script     1.1.1      Az.NotificationHubs                 {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificationHubListKey, Get-AzNotificationHubPNSCredential...}
Script     2.3.0      Az.OperationalInsights              {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsCustomLogDataSource, Disable-AzOperationalInsightsLinuxCustomLogCollection, Disable-AzOperationalInsightsIISLogCollection...}
Script     1.3.1      Az.PolicyInsights                   {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemediation...}
Script     1.1.2      Az.PowerBIEmbedded                  {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollectionAccessKey, Get-AzPowerBIWorkspace...}
Script     1.0.3      Az.PrivateDns                       {Get-AzPrivateDnsZone, Remove-AzPrivateDnsZone, Set-AzPrivateDnsZone, New-AzPrivateDnsZone...}
Script     3.0.1      Az.RecoveryServices                 {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecoveryServicesVaultSettingsFile, New-AzRecoveryServicesVault...}
Script     1.4.0      Az.RedisCache                       {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCachePatchSchedule, New-AzRedisCachePatchSchedule...}
Script     1.0.3      Az.Relay                            {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-AzRelayNamespace...}
Script     3.0.1      Az.Resources                        {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment...}
Script     1.4.1      Az.ServiceBus                       {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNamespace, Remove-AzServiceBusNamespace...}
Script     2.2.0      Az.ServiceFabric                    {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricClusterCertificate, Add-AzServiceFabricNode, Add-AzServiceFabricNodeType...}
Script     1.2.0      Az.SignalR                          {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey...}
Script     2.12.0     Az.Sql                              {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEncryptionActivity, Set-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseUpgradeHint...}
Script     1.1.0      Az.SqlVirtualMachine                {New-AzSqlVM, Get-AzSqlVM, Update-AzSqlVM, Remove-AzSqlVM...}
Script     3.0.0      Az.Storage                          {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey...}
Script     1.3.0      Az.StorageSync                      {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzStorageSyncService, Set-AzStorageSyncService...}
Script     1.0.1      Az.StreamAnalytics                  {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefinition, New-AzStreamAnalyticsFunction, Remove-AzStreamAnalyticsFunction...}
Script     1.0.0      Az.Support                          {Get-AzSupportService, Get-AzSupportProblemClassification, Get-AzSupportTicket, Get-AzSupportTicketCommunication...}
Script     1.0.4      Az.TrafficManager                   {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFromEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Remove-AzTrafficManagerCustomHeaderFromProfile...}
Script     2.1.0      Az.Websites                         {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppServicePlan...}
Script     1.0.1      Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation}
Binary     1.0.0.1    PackageManagement                   {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...}
Script     3.4.0      Pester                              {Describe, Context, It, Should...}
Manifest   2.3.0      Posh-SSH                            {Get-SCPFile, Get-SCPFolder, Get-SCPItem, Get-SFTPFile...}
Script     1.0.0.1    PowerShellGet                       {Install-Module, Find-Module, Save-Module, Update-Module...}
Script     2.0.0      PSReadline                          {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHandler, Get-PSReadLineOption...}

I tried with different older versions too, as some systems where i tried it on, didn't have the most recent Az packages installed

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Drewm3, @avirishuv.

Issue Details

---

## Description Recently one of my automated test scripts for a web application started failing constantly, after working fine for 2-3 years (I once migrated from the old AzureRM namespace). The script does the following steps: 1. Setup VM (Image Windows2019-datacenter-latest) 2. Add IIS, Enable WinRM 3. Restart AzVM (via Restart-AzVM -ResourceGroupName $resourceGroup -Name $vmName) 4. Run a few operations via WinRM, that takes like 20 minutes As of a few weeks (maybe 2-3 weeks) ago this worked fine, and the Restart behaviour worked synchronously and it more or less immediately restarted the VM and busily waited until the restart was done and step 4 worked fine As of now it apparently reboots in the middle of my Step 4 and the script aborts with connection loss errors. The Azure Portal Activity log for that machine lists a Restart VM entry at the correct time, but the actual Event log in the system with the appropriate system events about stopped services is like 5 minutes later. I've now removed the Restart step from my script (it's not necessary anymore, maybe it was necessary back in the win2016 image times) and my scripts works again, so I currently conclude that some behaviour change happened to Restart-AzVM. I currently have a few ideas what might be the culprit: 1. The behaviour of Restart-AzVM changed (probably on the server side, as it does not work with older Az Versions either) 2. The behaviour of the image changed, so it does not reboot immediately anymore but instead waits for some timeout (you know like Desktop and that "Open applications preventing reboot dialog"). 3. My usage of Restart AzVM was bonkers and I should have never used it without some active polling to tell me when it actually restarted and it was sheer luck that it worked. I kinda think it's something like 2 where e.g Windows Update running on the system is now preventing the reboot until it's done and previously was just working because nothing blocked. ## Environment data ``` Name Value ---- ----- PSVersion 5.1.19041.610 PSEdition Desktop PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...} BuildVersion 10.0.19041.610 CLRVersion 4.0.30319.42000 WSManStackVersion 3.0 PSRemotingProtocolVersion 2.3 SerializationVersion 1.1.0.1 ``` I also tried with the most recent 7.1, but no difference ## Module versions ```powershell ModuleType Version Name ExportedCommands ---------- ------- ---- ---------------- Script 2.2.1 Az.Accounts {Disable-AzDataCollection, Disable-AzContextAutosave, Enable-AzDataCollection, Enable-AzContextAutosave...} Script 1.1.1 Az.Advisor {Get-AzAdvisorRecommendation, Enable-AzAdvisorRecommendation, Disable-AzAdvisorRecommendation, Get-AzAdvisorConfiguration...} Script 2.0.1 Az.Aks {Get-AzAksCluster, New-AzAksCluster, Remove-AzAksCluster, Import-AzAksCredential...} Script 1.1.4 Az.AnalysisServices {Resume-AzAnalysisServicesServer, Suspend-AzAnalysisServicesServer, Get-AzAnalysisServicesServer, Remove-AzAnalysisServicesServer...} Script 2.1.0 Az.ApiManagement {Add-AzApiManagementApiToGateway, Add-AzApiManagementApiToProduct, Add-AzApiManagementProductToGroup, Add-AzApiManagementRegion...} Script 1.0.0 Az.AppConfiguration {Get-AzAppConfigurationStore, Get-AzAppConfigurationStoreKey, New-AzAppConfigurationStore, New-AzAppConfigurationStoreKey...} Script 1.1.0 Az.ApplicationInsights {Get-AzApplicationInsights, New-AzApplicationInsights, Remove-AzApplicationInsights, Update-AzApplicationInsights...} Script 1.4.0 Az.Automation {Get-AzAutomationHybridWorkerGroup, Remove-AzAutomationHybridWorkerGroup, Get-AzAutomationJobOutputRecord, Import-AzAutomationDscNodeConfiguration...} Script 3.1.0 Az.Batch {Remove-AzBatchAccount, Get-AzBatchAccount, Get-AzBatchAccountKey, New-AzBatchAccount...} Script 2.0.0 Az.Billing {Get-AzBillingInvoice, Get-AzBillingPeriod, Get-AzEnrollmentAccount, Get-AzConsumptionBudget...} Script 1.6.0 Az.Cdn {Get-AzCdnProfile, Get-AzCdnProfileSsoUrl, New-AzCdnProfile, Remove-AzCdnProfile...} Script 1.8.0 Az.CognitiveServices {Get-AzCognitiveServicesAccount, Get-AzCognitiveServicesAccountKey, Get-AzCognitiveServicesAccountSku, Get-AzCognitiveServicesAccountType...} Script 4.6.0 Az.Compute {Remove-AzAvailabilitySet, Get-AzAvailabilitySet, New-AzAvailabilitySet, Update-AzAvailabilitySet...} Script 1.0.3 Az.ContainerInstance {New-AzContainerGroup, Get-AzContainerGroup, Remove-AzContainerGroup, Get-AzContainerInstanceLog} Script 2.0.0 Az.ContainerRegistry {New-AzContainerRegistry, Get-AzContainerRegistry, Update-AzContainerRegistry, Remove-AzContainerRegistry...} Script 1.1.0 Az.DataBoxEdge {Get-AzDataBoxEdgeJob, Get-AzDataBoxEdgeDevice, Invoke-AzDataBoxEdgeDevice, New-AzDataBoxEdgeDevice...} Script 1.0.1 Az.Databricks {Get-AzDatabricksVNetPeering, Get-AzDatabricksWorkspace, New-AzDatabricksVNetPeering, New-AzDatabricksWorkspace...} Script 1.11.1 Az.DataFactory {Set-AzDataFactoryV2, Update-AzDataFactoryV2, Get-AzDataFactoryV2, Remove-AzDataFactoryV2...} Script 1.0.2 Az.DataLakeAnalytics {Get-AzDataLakeAnalyticsDataSource, New-AzDataLakeAnalyticsCatalogCredential, Remove-AzDataLakeAnalyticsCatalogCredential, Set-AzDataLakeAnalyticsCatalogCredential...} Script 1.3.0 Az.DataLakeStore {Get-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreTrustedIdProvider, Remove-AzDataLakeStoreFirewallRule, Set-AzDataLakeStoreTrustedIdProvider...} Script 1.0.0 Az.DataShare {New-AzDataShareAccount, Get-AzDataShareAccount, Remove-AzDataShareAccount, New-AzDataShare...} Script 1.1.0 Az.DeploymentManager {Get-AzDeploymentManagerArtifactSource, New-AzDeploymentManagerArtifactSource, Set-AzDeploymentManagerArtifactSource, Remove-AzDeploymentManagerArtifactSource...} Script 2.0.1 Az.DesktopVirtualization {Disconnect-AzWvdUserSession, Expand-AzWvdMsixImage, Get-AzWvdApplication, Get-AzWvdApplicationGroup...} Script 1.0.2 Az.DevTestLabs {Get-AzDtlAllowedVMSizesPolicy, Get-AzDtlAutoShutdownPolicy, Get-AzDtlAutoStartPolicy, Get-AzDtlVMsPerLabPolicy...} Script 1.1.2 Az.Dns {Get-AzDnsRecordSet, New-AzDnsRecordConfig, Remove-AzDnsRecordSet, Set-AzDnsRecordSet...} Script 1.3.0 Az.EventGrid {New-AzEventGridTopic, Get-AzEventGridTopic, Set-AzEventGridTopic, New-AzEventGridTopicKey...} Script 1.7.1 Az.EventHub {New-AzEventHubNamespace, Get-AzEventHubNamespace, Set-AzEventHubNamespace, Remove-AzEventHubNamespace...} Script 1.6.1 Az.FrontDoor {New-AzFrontDoor, Get-AzFrontDoor, Set-AzFrontDoor, Remove-AzFrontDoor...} Script 2.0.0 Az.Functions {Get-AzFunctionApp, Get-AzFunctionAppAvailableLocation, Get-AzFunctionAppPlan, Get-AzFunctionAppSetting...} Script 4.1.0 Az.HDInsight {Get-AzHDInsightJob, New-AzHDInsightSqoopJobDefinition, Wait-AzHDInsightJob, New-AzHDInsightStreamingMapReduceJobDefinition...} Script 1.1.0 Az.HealthcareApis {New-AzHealthcareApisService, Remove-AzHealthcareApisService, Set-AzHealthcareApisService, Get-AzHealthcareApisService} Script 2.7.0 Az.IotHub {Add-AzIotHubKey, Get-AzIotHubEventHubConsumerGroup, Get-AzIotHubConnectionString, Get-AzIotHubJob...} Script 3.1.0 Az.KeyVault {Add-AzManagedHsmKey, Get-AzManagedHsmKey, Remove-AzManagedHsmKey, Undo-AzManagedHsmKeyRemoval...} Script 1.0.0 Az.Kusto {Add-AzKustoClusterLanguageExtension, Add-AzKustoDatabasePrincipal, Get-AzKustoAttachedDatabaseConfiguration, Get-AzKustoCluster...} Script 1.4.0 Az.LogicApp {Get-AzIntegrationAccountAgreement, Get-AzIntegrationAccountAssembly, Get-AzIntegrationAccountBatchConfiguration, Get-AzIntegrationAccountCallbackUrl...} Script 1.1.3 Az.MachineLearning {Move-AzMlCommitmentAssociation, Get-AzMlCommitmentAssociation, Get-AzMlCommitmentPlanUsageHistory, Remove-AzMlCommitmentPlan...} Script 1.1.0 Az.Maintenance {Get-AzApplyUpdate, Get-AzConfigurationAssignment, Get-AzMaintenanceConfiguration, Get-AzMaintenanceUpdate...} Script 2.0.0 Az.ManagedServices {Get-AzManagedServicesAssignment, New-AzManagedServicesAssignment, Remove-AzManagedServicesAssignment, Get-AzManagedServicesDefinition...} Script 1.0.2 Az.MarketplaceOrdering {Get-AzMarketplaceTerms, Set-AzMarketplaceTerms} Script 1.1.1 Az.Media {Sync-AzMediaServiceStorageKey, Set-AzMediaServiceKey, Get-AzMediaServiceKey, Get-AzMediaServiceNameAvailability...} Script 2.2.0 Az.Monitor {Get-AzMetricDefinition, Get-AzMetric, Remove-AzLogProfile, Get-AzLogProfile...} Script 4.3.0 Az.Network {Add-AzApplicationGatewayAuthenticationCertificate, Get-AzApplicationGatewayAuthenticationCertificate, New-AzApplicationGatewayAuthenticationCertificate, Remove-AzApplicationGatewayAuthenticationCertificate...} Script 1.1.1 Az.NotificationHubs {Get-AzNotificationHub, Get-AzNotificationHubAuthorizationRule, Get-AzNotificationHubListKey, Get-AzNotificationHubPNSCredential...} Script 2.3.0 Az.OperationalInsights {New-AzOperationalInsightsAzureActivityLogDataSource, New-AzOperationalInsightsCustomLogDataSource, Disable-AzOperationalInsightsLinuxCustomLogCollection, Disable-AzOperationalInsightsIISLogCollection...} Script 1.3.1 Az.PolicyInsights {Get-AzPolicyEvent, Get-AzPolicyState, Get-AzPolicyStateSummary, Get-AzPolicyRemediation...} Script 1.1.2 Az.PowerBIEmbedded {Remove-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollection, Get-AzPowerBIWorkspaceCollectionAccessKey, Get-AzPowerBIWorkspace...} Script 1.0.3 Az.PrivateDns {Get-AzPrivateDnsZone, Remove-AzPrivateDnsZone, Set-AzPrivateDnsZone, New-AzPrivateDnsZone...} Script 3.0.1 Az.RecoveryServices {Get-AzRecoveryServicesBackupProperty, Get-AzRecoveryServicesVault, Get-AzRecoveryServicesVaultSettingsFile, New-AzRecoveryServicesVault...} Script 1.4.0 Az.RedisCache {Remove-AzRedisCachePatchSchedule, New-AzRedisCacheScheduleEntry, Get-AzRedisCachePatchSchedule, New-AzRedisCachePatchSchedule...} Script 1.0.3 Az.Relay {New-AzRelayNamespace, Get-AzRelayNamespace, Set-AzRelayNamespace, Remove-AzRelayNamespace...} Script 3.0.1 Az.Resources {Get-AzProviderOperation, Remove-AzRoleAssignment, Get-AzRoleAssignment, New-AzRoleAssignment...} Script 1.4.1 Az.ServiceBus {New-AzServiceBusNamespace, Get-AzServiceBusNamespace, Set-AzServiceBusNamespace, Remove-AzServiceBusNamespace...} Script 2.2.0 Az.ServiceFabric {Add-AzServiceFabricClientCertificate, Add-AzServiceFabricClusterCertificate, Add-AzServiceFabricNode, Add-AzServiceFabricNodeType...} Script 1.2.0 Az.SignalR {New-AzSignalR, Get-AzSignalR, Get-AzSignalRKey, New-AzSignalRKey...} Script 2.12.0 Az.Sql {Get-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseTransparentDataEncryptionActivity, Set-AzSqlDatabaseTransparentDataEncryption, Get-AzSqlDatabaseUpgradeHint...} Script 1.1.0 Az.SqlVirtualMachine {New-AzSqlVM, Get-AzSqlVM, Update-AzSqlVM, Remove-AzSqlVM...} Script 3.0.0 Az.Storage {Get-AzStorageAccount, Get-AzStorageAccountKey, New-AzStorageAccount, New-AzStorageAccountKey...} Script 1.3.0 Az.StorageSync {Invoke-AzStorageSyncCompatibilityCheck, New-AzStorageSyncService, Get-AzStorageSyncService, Set-AzStorageSyncService...} Script 1.0.1 Az.StreamAnalytics {Get-AzStreamAnalyticsFunction, Get-AzStreamAnalyticsDefaultFunctionDefinition, New-AzStreamAnalyticsFunction, Remove-AzStreamAnalyticsFunction...} Script 1.0.0 Az.Support {Get-AzSupportService, Get-AzSupportProblemClassification, Get-AzSupportTicket, Get-AzSupportTicketCommunication...} Script 1.0.4 Az.TrafficManager {Add-AzTrafficManagerCustomHeaderToEndpoint, Remove-AzTrafficManagerCustomHeaderFromEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, Remove-AzTrafficManagerCustomHeaderFromProfile...} Script 2.1.0 Az.Websites {Get-AzAppServicePlan, Set-AzAppServicePlan, New-AzAppServicePlan, Remove-AzAppServicePlan...} Script 1.0.1 Microsoft.PowerShell.Operation.V... {Get-OperationValidation, Invoke-OperationValidation} Binary 1.0.0.1 PackageManagement {Find-Package, Get-Package, Get-PackageProvider, Get-PackageSource...} Script 3.4.0 Pester {Describe, Context, It, Should...} Manifest 2.3.0 Posh-SSH {Get-SCPFile, Get-SCPFolder, Get-SCPItem, Get-SFTPFile...} Script 1.0.0.1 PowerShellGet {Install-Module, Find-Module, Save-Module, Update-Module...} Script 2.0.0 PSReadline {Get-PSReadLineKeyHandler, Set-PSReadLineKeyHandler, Remove-PSReadLineKeyHandler, Get-PSReadLineOption...} ``` I tried with different older versions too, as some systems where i tried it on, didn't have the most recent Az packages installed

Author:	Tornhoof
Assignees:	-
Labels:	`Compute - VM`, `Service Attention`, `customer-reported`, `question`
Milestone:	-

[dingmeng-xue]

Compute team, please help to look into this question.

[axayjo]

@avirishuv can you please take a look at this issue?

[avirishuv]

Investigating on the API side as well on the PS side, and will provide update. cc @bilaakpan-ms

[avirishuv]

hi @Tornhoof update on this issue: there is a bug we have identified on the server side which is likely the reason you are seeing the issue. The fix is currently in the backlog and we'll keep you posted about its progress/rollout.

[avirishuv]

quick update: the fix is still in the backlog and planned to be worked on over the next few weeks. Apologies for the delay here.

[petermicuch]

Hi @avirishuv. I guess this does not influence only Restart-AzVM but also other commands like Invoke-AzVMRunCommand. I am seeing the same issue as @Tornhoof described and it is close to impossible to do some automation with Azure VMs. Do you have some ETA for this to be fixed?

[avirishuv]

@petermicuch thanks for reporting, let me check and get back to you.

[petermicuch]

@avirishuv this might have been my mistake. I had a bug in script. After I fixed it, the Invoke-AzVMRunCommand seems to work fine. Restarting of machine in my case now also works fine...I started to capture return value from Restart-AzVM, but that should hopefully not have influence on the behavior.

[avirishuv]

thanks for the update @petermicuch , good to know that the Invoke-AzVMRunCommand is working fine.

[petermicuch]

@avirishuv is there any workaround to find out that VM finished restarting until we get a fix? It happened to me many times, that Restart-AzVM returned, but machine has not yet restarted at all. So when I execute next command with Invoke-AzVMRunCommand, that one gets interrupted. I.e. in my example I am installing domain services on the VM and after restart, I should see different fqdn, but I do not, since the restart did not happened even though Restart-AzVM returns.

[avirishuv]

@petermicuch checking for the power state of the VM before you trigger the Invoke-AzVMRunCommand could be one possible workaround. If the VM has reached "Running" state then restart has completed, if not wait for some more time before re-checking.

[petermicuch]

@avirishuv actually that does not work, since VM is in state running and only restarts later as also @Tornhoof described. Since in this special case I installed domain controller, I just check if the FQDN is full domain name. By that I know that restart has completed, because before that, FQDN does not contain domain yet. Other possible solution would be to have script running on startup, that sets some environment variable always and you just reset it before restart. But the best would be to have final fix from you :-).

[michaelatvaronis]

@avirishuv any update on rolling out a fix? this is a serious show stopper, we've experienced too many Restart-AzVM operations where the VM actually performs restart only 1h later!

joining to @petermicuch question above, is there any workaround to find out that VM finished restarting until we get a fix?

[avirishuv]

quick note: the fix for this is currently in progress, I'll provide an update here once its rolled out.

[Viajaz]

@avirishuv Has the fix been rolled out?

[avirishuv]

hi @Viajaz fix rollout is in progress, I'll confirm once it has completed roll out to all regions.

[brwilkinson]

Following issue...

[lpalerm]

Following issue

[nathanmitten]

Has the fix been rolled out? We are still seeing some issues with the command.

[blevenson-kcare]

Still appears to be an issue. @Viajaz, @avirishuv , any update?

[avirishuv]

The fix is currently being rolled out to production regions, will update once this is completed.

[navba-MSFT]

@Tornhoof Apologies for the late reply. We are looking into this issue and we will update this thread once we have more details.

@avirishuv Could you please provide an update on this issue ? Awaiting your reply.

[JeffZe-MSFT]

@avirishuv Do you have any updates?

[avirishuv]

The fix has been rolled out. Closing the issue now, please feel free to reopen if you have any additional questions on this topic.

[rstra]

what version was this resolved in?