adding additional Networks to LabConfig assignments getting wild

[Karl-WE]

Dear Jaromir,

I am using the following MSLabs Config

1..2 | ForEach-Object {$LABConfig.VMs += @{ VMName = "ASNode$_" ; Configuration = 'S2D' ; ParentVHD = 'AzSHCI22H2_G2.vhdx' ; SSDNumber = 5 ; SSDSize= 2TB ; MemoryStartupBytes= 6144MB; VMProcessorCount=12 ; vTPM=$true ; Unattend="NoDjoin" ; NestedVirt=$true ; AdditionalNetworks=$true }
#region additional networks

         $LABConfig.AdditionalNetworksConfig += @{ 
                NetName = 'SMB1';                            # Network Name
                NetAddress='172.16.1.';                      # Network Addresses prefix. (starts with 1), therefore first VM with Additional network config will have IP 172.16.1.1
                NetVLAN='711';                                 # VLAN tagging
                Subnet='255.255.255.0'                       # Subnet Mask
        }

        $LABConfig.AdditionalNetworksConfig += @{ 
                NetName = 'SMB2';                        # Network Name
                NetAddress='172.16.2.';                      # Network Addresses prefix. (starts with 1), therefore first VM with Additional network config will have IP 172.16.1.1
                NetVLAN='712';                                 # VLAN tagging
                Subnet='255.255.255.0'                       # Subnet Mask
        }

    #endregion additional networks
}

Goal: The goal is to achive a lab, where management + compute and storage intends are separated as per https://learn.microsoft.com/en-us/azure-stack/hci/plan/two-node-switched-non-converged

Expected behaviour: Deploy.ps1 should create 2 AzNodes, with 2 Management NICs and 2 SMB NICs

Result ASNode1 Ethernet = Management 1, IP 10.0.0.17 Ethernet 2 = Management 2 10.0.0.18 Ethernet 3 = SMB 1 172.16.1.1 Ethernet 4 = SMB 2 172.16.2.1

Actual behaviour: Deploy.ps1 creates 2 AzNodes, with 2 Management NICs and 4 SMB* NICs In addition the second SMB1 / SMB2 does not get any IPv4 assigned as expected.

The Management Adapters are not created before the additional networks and the IP does not match the lowest IP on the first vNIC anymore.

Result Ethernet = SMB 2 172.16.2.1 Ethernet 2 = Management 2, IP 10.0.0.18 Ethernet 3 = Management 1, IP 10.0.0.17 Ethernet 4 = SMB 1 172.16.1.1

*to reduce confusion. in this example I have removed the secondary SMB1 / SMB2 NICs that would not have any 172.16. addresses.

If I remove additionalnetworksconfig, the order and IP assignment is as a expected.

Outputs: the IP to MAC address matches that the lowest MAC uses the lowest IP

the IP address and Adapter Name does not match with the order they were created, except I do not use additional networks.

VM view (appears correct, including VLAN assignment)

[Karl-WE]

Log output Node 1 Adding Adapter SMB1 with IP 172.16.1.1 Success Adding Adapter SMB2 with IP 172.16.2.1 Success Adding Adapter SMB1 with IP 172.16.1.1 Success Adding Adapter SMB2 with IP 172.16.2.1

Node 2 Configuring Additional networks Adding Adapter SMB1 with IP 172.16.1.2 Success Adding Adapter SMB2 with IP 172.16.2.2 Success Adding Adapter SMB1 with IP 172.16.1.2 Success Adding Adapter SMB2 with IP 172.16.2.2 Success

Logfile Deploy.log

[Karl-WE]

Why do I consider this to be a problem?

1. why the MSLab is creating 2 SMB NICs on each node, trying to assign the same IP Address (which will fail and leaving the second without any -> APIPA)

2. The IP and Netadapter Name assignment does not follow a predictable pattern and is not even the same on Node 1 and Node 2, so following Lab scripts would fail, when trying to use these template lab VMs with https://github.com/DellGEOS/AzureStackHOLs/blob/main/lab-guides/08-DeployAzureStackHCICluster-DeploymentTool, Task 5 Step 2

While of course altering the config in this step, to reflect the separate storage networks. In the default the scenario is fully converged.

"HostNetwork": {
                    "Intents": [
                        {
                            "Name": "Compute_Management",
                            "TrafficType": [
                                "Compute",
                                "Management"
                            ],
                            "Adapter": [
                                "Ethernet",
                                "Ethernet 2"
                            ],
                            "OverrideVirtualSwitchConfiguration": false,
                            "VirtualSwitchConfigurationOverrides": {
                                "EnableIov": "",
                                "LoadBalancingAlgorithm": ""
                            },
                            "OverrideQoSPolicy": false,
                            "QoSPolicyOverrides": {
                                "PriorityValue8021Action_Cluster": "",
                                "PriorityValue8021Action_SMB": "",
                                "BandwidthPercentage_SMB": ""
                            },
                            "OverrideAdapterProperty": true,
                            "AdapterPropertyOverrides": {
                                "NetworkDirect": "0",
                            }
                        }
{
                            "Name": "Storage",
                            "TrafficType": [
                                "Storage"
                            ],
                            "Adapter": [
                                "Ethernet 3",
                                "Ethernet 4"
                            ],
                            "OverrideVirtualSwitchConfiguration": false,
                            "VirtualSwitchConfigurationOverrides": {
                                "EnableIov": "",
                                "LoadBalancingAlgorithm": ""
                            },
                            "OverrideQoSPolicy": false,
                            "QoSPolicyOverrides": {
                                "PriorityValue8021Action_Cluster": "",
                                "PriorityValue8021Action_SMB": "",
                                "BandwidthPercentage_SMB": ""
                            },
                            "OverrideAdapterProperty": true,
                            "AdapterPropertyOverrides": {
                                "NetworkDirect": "0",
                            }
                        }
                    ],
                    "StorageNetworks": [
                        {
                            "Name": "SMB1",
                            "NetworkAdapterName": "Ethernet 3",
                            "VlanId": 711
                        },
                        {
                            "Name": "SMB2",
                            "NetworkAdapterName": "Ethernet 4",
                            "VlanId": 712
                        }
                    ]

[jaromirk]

You are overcomplicating it :) ... Here is the example you need https://github.com/microsoft/MSLab/blob/master/Scenarios/AzSHCI%20and%20NetATC/LabConfig.ps1

[Karl-WE]

Thank you for the hint. Also that I have learned about the min of 4 vCores for RSS. Interesting. Have not read about it in the docs.

Could you be bothered to look into the reported issue, as it should be reproducible and I believe there is a problem with the counter in the loop or the order when things are created.

Thank you, it will help not only my scenario.

The NetATC scenario might not be fitting or I need to combine it with the one using Microsoft cloud deployment kit.

I will give it a try today.

[Karl-WE]

"Your labconfig is wrong. You kept additional networks in 1..2 loop. It has to be separate."

1..2 | ForEach-Object {$LABConfig.VMs += @{ VMName = "ASNode$_" ; Configuration = 'S2D' ; ParentVHD = 'AzSHCI22H2_G2.vhdx' ; SSDNumber = 5 ; SSDSize= 2TB ; MemoryStartupBytes= 6144MB; VMProcessorCount=12 ; vTPM=$true ; Unattend="NoDjoin" ; NestedVirt=$true ; AdditionalNetworks=$true }}
#region additional networks

$LABConfig.AdditionalNetworksConfig += @{ 
                NetName = 'SMB1';                            # Network Name
                NetAddress='172.16.1.';                      # Network Addresses prefix. (starts with 1), therefore first VM with Additional network config will have IP 172.16.1.1
                NetVLAN='711';                                 # VLAN tagging
                Subnet='255.255.255.0'                       # Subnet Mask
        }

$LABConfig.AdditionalNetworksConfig += @{ 
                NetName = 'SMB2';                        # Network Name
                NetAddress='172.16.2.';                      # Network Addresses prefix. (starts with 1), therefore first VM with Additional network config will have IP 172.16.1.1
                NetVLAN='712';                                 # VLAN tagging
                Subnet='255.255.255.0'                       # Subnet Mask
        }

    #endregion additional networks

like that?

[jaromirk]

I need to get into PC :). Ill send you example and will test it. I think it's wrong. Need to validate. :). Here is scenario where I use it. https://github.com/microsoft/MSLab/blob/master/Scenarios/AzSHCI%20and%20Stretch%20Cluster/LabConfig.ps1

[jaromirk]

can you include entire config pls?

[Karl-WE]

Hi Jaromir, putting the additional network configs outside the for each loop did the trick, well partly*, and solves the problem (Layer 8) of duplicate NICs.

*Still the order of NetworkAdapter name and IP does not match on both nodes, which would cause subsequent problems for scenarios, MS Cloud Deployment Toolkit and eventually even NetworkATC.

I really wonder where this comes from, the cause must be in the order how NICs are added to the VM

Expected order with increasing MAC Addresses Node 1

26 = Ethernet, Management 1, IP 10.0.0.12
27 = Ethernet #2, Management 2, IP 10.0.0.13
28 = Ethernet #3, IP 172.16.1.1
29 = Ethernet #4, IP 172.16.2.1

Node 2
2A = Ethernet, Management 1, IP 10.0.0.14
2B = Ethernet #2, Management 2, IP 10.0.0.15
2C = Ethernet #3, SMB 1, IP 172.16.1.2
2D = Ethernet #4, SMB 2, IP 172.16.2.2

LabConfig.zip

Watch out on the screenshot i Node 2 is left Node 1 is on the right.

[Karl-WE]

@jaromirk know you are busy. Have you had any time to understand what is happening?