Failing upgrade when Inbound NAT Pools configured

MattGammonAtSeeTickets commented 1 year ago

Hello There,

We are trying to upgrade a Load Balancer which is in front of a Virtual machine scale set of Pulse Secure Virtual Traffic Managers, We keep getting the following error;

2022-11-11 01:25:43.5808 [Error] - [BackupBasicLoadBalancer] An error occured while exporting the VMSS '' for backup purposes. Error: Exception calling "Serialize" with "3" argument(s): "The specified type Microsoft.Azure.Commands.Compute.Automation.Models.PSVirtualMachineScaleSet must derive from the specific value's type System.Object[]."

the PowerShell window keeps closing so not able to get any messages from that, tried to make sure all Modules are the most updated ones, we have two setups of the Virtual machine scale set with Load Balancer and both are failing to upgrade.

Thanks,

Matt G

mbrat2005 commented 1 year ago

Hi @MattGammonAtSeeTickets,

It looks like we are not getting a valid VMSS object ID from the list of backend pool members on your Basic load balancer--that error should have the VMSS name in the single quotes.

Would you mind sharing the JSON view of your Basic load balancer? The easiest way to get this is to navigate to your load balancer in the Portal, then on the Overview page, click the JSON View link at the top right. Feel free to scrub subscription IDs or other sensitive details.

MattGammonAtSeeTickets commented 1 year ago

Hi @mbrat2005

Thanks, below are the JSON you have requested from one we are testing with at the momnet;

{ "name": "mmtestdel-vtmLB", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers", "location": "*******", "properties": { "provisioningState": "Succeeded", "resourceGuid": "d2c9e6d6-4749-4c7e-a197-6f2a4456288d", "frontendIPConfigurations": [ { "name": "LoadBalancerFrontEnd", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers/frontendIPConfigurations", "properties": { "provisioningState": "Succeeded", "privateIPAllocationMethod": "Dynamic", "publicIPAddress": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/publicIPAddresses/mmtestdel" }, "loadBalancingRules": [ { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/loadBalancingRules/vTMTrafficRule" } ], "inboundNatRules": [ { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/sshNatPool.0" }, { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/adminNatPool.0" }, { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/restNatPool.0" }, { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/martindeletesshnonat" } ], "inboundNatPools": [ { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatPools/sshNatPool" }, { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatPools/adminNatPool" }, { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatPools/restNatPool" } ] } } ], "backendAddressPools": [ { "name": "LoadBalancerBackend", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/backendAddressPools/LoadBalancerBackend", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "properties": { "provisioningState": "Succeeded", "backendIPConfigurations": [ { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Compute/virtualMachineScaleSets/mmtestdel-vmss/virtualMachines/0/networkInterfaces/mmtestdel-nic/ipConfigurations/mmtestdel-ipconfig" } ], "loadBalancingRules": [ { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/loadBalancingRules/vTMTrafficRule" } ] }, "type": "Microsoft.Network/loadBalancers/backendAddressPools" } ], "loadBalancingRules": [ { "name": "vTMTrafficRule", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/loadBalancingRules/vTMTrafficRule", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers/loadBalancingRules", "properties": { "provisioningState": "Succeeded", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" }, "frontendPort": 443, "backendPort": 443, "enableFloatingIP": false, "idleTimeoutInMinutes": 4, "protocol": "Tcp", "loadDistribution": "Default", "backendAddressPool": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/backendAddressPools/LoadBalancerBackend" }, "probe": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/probes/vtmAdminProbe" } } } ], "probes": [ { "name": "vtmAdminProbe", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/probes/vtmAdminProbe", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "properties": { "provisioningState": "Succeeded", "protocol": "Tcp", "port": 9090, "intervalInSeconds": 5, "numberOfProbes": 2, "loadBalancingRules": [ { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/loadBalancingRules/vTMTrafficRule" } ] }, "type": "Microsoft.Network/loadBalancers/probes" } ], "inboundNatRules": [ { "name": "sshNatPool.0", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/sshNatPool.0", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers/inboundNatRules", "properties": { "provisioningState": "Succeeded", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" }, "frontendPort": 50000, "backendPort": 22, "enableFloatingIP": false, "idleTimeoutInMinutes": 4, "protocol": "Tcp", "backendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Compute/virtualMachineScaleSets/mmtestdel-vmss/virtualMachines/0/networkInterfaces/mmtestdel-nic/ipConfigurations/mmtestdel-ipconfig" } } }, { "name": "adminNatPool.0", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/adminNatPool.0", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers/inboundNatRules", "properties": { "provisioningState": "Succeeded", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" }, "frontendPort": 50100, "backendPort": 9090, "enableFloatingIP": false, "idleTimeoutInMinutes": 4, "protocol": "Tcp", "backendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Compute/virtualMachineScaleSets/mmtestdel-vmss/virtualMachines/0/networkInterfaces/mmtestdel-nic/ipConfigurations/mmtestdel-ipconfig" } } }, { "name": "restNatPool.0", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/restNatPool.0", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers/inboundNatRules", "properties": { "provisioningState": "Succeeded", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" }, "frontendPort": 50200, "backendPort": 9070, "enableFloatingIP": false, "idleTimeoutInMinutes": 4, "protocol": "Tcp", "backendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Compute/virtualMachineScaleSets/mmtestdel-vmss/virtualMachines/0/networkInterfaces/mmtestdel-nic/ipConfigurations/mmtestdel-ipconfig" } } }, { "name": "martindeletesshnonat", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatRules/martindeletesshnonat", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "type": "Microsoft.Network/loadBalancers/inboundNatRules", "properties": { "provisioningState": "Succeeded", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" }, "frontendPort": 22, "backendPort": 22, "enableFloatingIP": false, "idleTimeoutInMinutes": 4, "protocol": "Tcp" } } ], "inboundNatPools": [ { "name": "sshNatPool", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatPools/sshNatPool", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "properties": { "provisioningState": "Succeeded", "frontendPortRangeStart": 50000, "frontendPortRangeEnd": 50063, "backendPort": 22, "protocol": "Tcp", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" } }, "type": "Microsoft.Network/loadBalancers/inboundNatPools" }, { "name": "adminNatPool", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatPools/adminNatPool", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "properties": { "provisioningState": "Succeeded", "frontendPortRangeStart": 50100, "frontendPortRangeEnd": 50163, "backendPort": 9090, "protocol": "Tcp", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" } }, "type": "Microsoft.Network/loadBalancers/inboundNatPools" }, { "name": "restNatPool", "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/inboundNatPools/restNatPool", "etag": "W/\"cdf91a38-6e8c-43e0-a717-57573e162eab\"", "properties": { "provisioningState": "Succeeded", "frontendPortRangeStart": 50200, "frontendPortRangeEnd": 50263, "backendPort": 9070, "protocol": "Tcp", "frontendIPConfiguration": { "id": "/subscriptions/********-****-****-****-************/resourceGroups/martin-test-delete/providers/Microsoft.Network/loadBalancers/mmtestdel-vtmLB/frontendIPConfigurations/LoadBalancerFrontEnd" } }, "type": "Microsoft.Network/loadBalancers/inboundNatPools" } ] }, "sku": { "name": "Basic" } }

mbrat2005 commented 1 year ago

Thank you @MattGammonAtSeeTickets for the details!

I'm having trouble reproducing the issue from your provided LB config. The problem appears to be that we are failing to extract a VMSS name and/or resource group from your Basic LB's backend address pool membership, but the value in your above sample is correctly formatted. The Get-AzVMSS function then accepts a blank value for the -VMScaleSetName and -ResourceGroupName parameter (which it should not do!) and returns all VMSSes, and we are unable to serialize the array value.

Can you please confirm the version of PowerShell you are running by executing $PSVersionTable.PSVersion?

If you want to get more into the weeds, you can modify the BackupBasicLoadBalancer.psm1 file within your downloaded module, replacing the section starting at line 95 with the code below, which will add more logging. If you do this, make sure to run Remove-Module AzureBasicLoadBalancerUpgrade before retrying the operation--then please provide the logged output:

# Backup VMSS Object
    $vmssIds = $BasicLoadBalancer.BackendAddressPools.BackendIpConfigurations.id | Foreach-Object { $_.split("virtualMachines")[0] } | Select-Object -Unique
    foreach ($vmssId in $vmssIds) {

        $vmssName = $vmssId.split("/")[8]
        $vmssRg = $vmssId.Split('/')[4]

        $message = "[BackupBasicLoadBalancer] Exporting VMSS with ID '$vmssId'. vmssName: '$vmssName', vmssRg: '$vmssRg'"
        log -Severity Information -Message $message

        try {

MattGammonAtSeeTickets commented 1 year ago

Hi @mbrat2005,

I have tried using both PowerShell 5.1 and 7.3 on my works PC,

not sure if done the logging right?; ` 2022-11-14 04:44:45.0650 [Information] - ############################## Initializing Start-AzBasicLoadBalancerUpgrade ############################## 2022-11-14 04:44:45.0783 [Information] - [Start-AzBasicLoadBalancerUpgrade] Checking that user is signed in to Azure PowerShell 2022-11-14 04:44:45.1930 [Information] - [Start-AzBasicLoadBalancerUpgrade] Loading Azure Resources 2022-11-14 04:44:45.2039 [Information] - [Start-AzBasicLoadBalancerUpgrade] Basic Load Balancer mmtestdel-vtmLB loaded 2022-11-14 04:44:45.2388 [Information] - [Test-SupportedMigrationScenario] Verifying if Load Balancer mmtestdel-vtmLB is valid for migration 2022-11-14 04:44:45.2428 [Information] - [Test-SupportedMigrationScenario] Verifying source load balancer SKU 2022-11-14 04:44:45.2528 [Information] - [Test-SupportedMigrationScenario] Source load balancer SKU is type Basic 2022-11-14 04:44:45.2558 [Information] - [Test-SupportedMigrationScenario] Checking if there are any backend pool members which are not virtualMachineScaleSets and that all backend pools are not empty 2022-11-14 04:44:45.2717 [Information] - [Test-SupportedMigrationScenario] All backend pools members virtualMachineScaleSets! 2022-11-14 04:44:45.2747 [Information] - [Test-SupportedMigrationScenario] Checking if there are more than one VMSS in the backend pool 2022-11-14 04:44:45.3096 [Information] - [Test-SupportedMigrationScenario] Basic Load Balancer has only one VMSS in the backend pool 2022-11-14 04:44:45.3146 [Information] - [Test-SupportedMigrationScenario] Checking that source load balancer is configured 2022-11-14 04:44:45.3226 [Information] - [Test-SupportedMigrationScenario] Load balancer has at least 1 frontend IP configuration 2022-11-14 04:44:45.3266 [Information] - [Test-SupportedMigrationScenario] Checking that standard load balancer name 'mmtestdel-vtmLB' 2022-11-14 04:44:45.6194 [Information] - [Test-SupportedMigrationScenario] Load balancer resource 'mmtestdel-vtmLB' is a Basic Load Balancer. The same name will be re-used. 2022-11-14 04:44:45.6294 [Information] - [Test-SupportedMigrationScenario] Checking if backend pools contain members which are members of another load balancer's backend pools... 2022-11-14 04:44:45.8359 [Information] - [Test-SupportedMigrationScenario] Checking for instances in backend pool member VMSS '' with Instance Protection configured 2022-11-14 04:44:45.8738 [Information] - [Test-SupportedMigrationScenario] No VMSS instances with Instance Protection found 2022-11-14 04:44:45.8807 [Information] - [Test-SupportedMigrationScenario] Determining if LB is internal or external based on FrontEndIPConfiguration[0]'s IP configuration 2022-11-14 04:44:45.8967 [Information] - [Test-SupportedMigrationScenario] FrontEndIPConfiguiration[0] is assigned a public IP address '/subscriptions/****----****/resourceGroups/martin-test-delete/providers/Microsoft.Network/publicIPAddresses/mmtestdel', so this LB is External 2022-11-14 04:44:45.8997 [Information] - [Test-SupportedMigrationScenario] Determining if there is a frontend IPV6 configuration 2022-11-14 04:44:46.0669 [Information] - [Test-SupportedMigrationScenario] Load Balancer mmtestdel-vtmLB is valid for migration 2022-11-14 04:44:46.0769 [Information] - [PublicLBMigration] Public Load Balancer Detected. Initiating Public Load Balancer Migration 2022-11-14 04:44:46.0908 [Information] - [GetVMSSFromBasicLoadBalancer] Initiating GetVMSSFromBasicLoadBalancer 2022-11-14 04:44:46.9601 [Information] - [GetVMSSFromBasicLoadBalancer] VMSS loaded Name mmtestdel-vmss vtmazuks-vmss from RG MARTIN-TEST-DELETE SEETICKETS-AZ-VTMS 2022-11-14 04:44:46.9850 [Information] - [BackupBasicLoadBalancer] Initiating Backup of Basic Load Balancer Configurations to path 'C:\Users\matt.gammon' 2022-11-14 04:44:47.1876 [Information] - [BackupBasicLoadBalancer] JSON backup Basic Load Balancer to file C:\Users\matt.gammon\State_mmtestdel-vtmLB_martin-test-delete_20221114T1644469890.json Completed 2022-11-14 04:44:47.1906 [Information] - [BackupBasicLoadBalancer] Exporting Basic Load Balancer ARM template to path 'C:\Users\matt.gammon'... 2022-11-14 04:44:48.0253 [Information] - [BackupBasicLoadBalancer] Completed export Basic Load Balancer ARM template to path 'C:\Users\matt.gammon\ARMTemplate_mmtestdel-vtmLB_martin-test-delete_20221114T1644469890.json'... 2022-11-14 04:44:48.2577 [Error] - [BackupBasicLoadBalancer] An error occured while exporting the VMSS '' for backup purposes. Error: Exception calling "Serialize" with "3" argument(s): "The specified type Microsoft.Azure.Commands.Compute.Automation.Models.PSVirtualMachineScaleSet must derive from the specific value's type System.Object[]."

`

mbrat2005 commented 1 year ago

Working with submitter directly...

mbrat2005 commented 1 year ago

An issue remains where the basic LB cannot be deleted because it fails with an error that the 'inboundNatPools' child resources are still in use by the VMSS. The VMSS is set to 'automatic' update policy

mbrat2005 commented 1 year ago

Good reference on Inbound NAT Pools and advantages of migrating to Inbound NAT Rules for VMSSes: Manage port forwarding for backend pool with Azure Load Balancer | Azure Blog and Updates | Microsoft Azure

mbrat2005 commented 1 year ago

Final issues fixed in PR 9

Azure / AzLoadBalancerMigration

Failing upgrade when Inbound NAT Pools configured #6