search

microsoft / hcsshim

Windows - Host Compute Service Shim
MIT License
570 stars 256 forks source link

Hns: POST after DELETE doesn't work without a delay #95

Open m-kostrzewa opened 7 years ago

m-kostrzewa commented 7 years ago

Deleting a HNS network and then creating another one immediately after doesn't work. Here's the error message:

Expected error:
    <*errors.errorString | 0xc0422f18b0>: {
        s: "HNS failed with error : Element not found. ",
    }
    HNS failed with error : Element not found.
not to have occurred

Here's a test case:

subnets1 := []hcsshim.Subnet{
    {
        AddressPrefix:  "172.100.0.0/20",
        GatewayAddress: "172.100.0.1",
    },
}
configuration1 := &hcsshim.HNSNetwork{
    Name:    "TestNetworkName1",
    Type:    "transparent",
    Subnets: subnets1,
}

subnets2 := []hcsshim.Subnet{
    {
        AddressPrefix:  "172.200.0.0/20",
        GatewayAddress: "172.200.0.1",
    },
}
configuration2 := &hcsshim.HNSNetwork{
    Name:    "TestNetworkName2",
    Type:    "transparent",
    Subnets: subnets2,
}

It("doesn't work if there's no delay after DELETE", func() {
    configBytes1, err := json.Marshal(configuration1)
    Expect(err).ToNot(HaveOccurred())
    response, err := hcsshim.HNSNetworkRequest("POST", "", string(configBytes1))
    Expect(err).ToNot(HaveOccurred())
    hnsID := response.Id

    _, err = hcsshim.HNSNetworkRequest("DELETE", hnsID, "")
    Expect(err).ToNot(HaveOccurred())

    //time.Sleep(time.Second * 20) // 20 second timeout "fixes" the issue

    configBytes2, err := json.Marshal(configuration2)
    Expect(err).ToNot(HaveOccurred())
    response, err = hcsshim.HNSNetworkRequest("POST", "", string(configBytes2))
    Expect(err).To(HaveOccurred()) // !!! ERROR

    _, err = hcsshim.HNSNetworkRequest("GET", hnsID, "")
    Expect(err).To(HaveOccurred()) // but can't GET the deleted network either
})

Note that sleeping for 20 seconds after deleting a network seems to "fix" the issue. 10 second timeout is not enough.

update: I repeated this in Powershell, so this may be problem with HNS. I posted an issue here: https://github.com/Microsoft/Virtualization-Documentation/issues/516

msabansal commented 7 years ago

This is probably because of a delay in applying bindings to the external network adapter when the transparent network is deleted. Will look into this

msabansal commented 7 years ago

I tried this on an RS2 build and it doesn't seem to be a problem. I will try reproducing this on Windows Server 2016.

PS C:\Users\Administrator> 1..10 | % { Remove-ContainerNetwork -name transparent -Force; New-ContainerNetwork -name tra nsparent -Mode Transparent; }

Name Id Subnets Mode SourceMac DNSServers DNSSuffix

transparent 53765384-11a1-42f7-ab32-28424273895f {} Transparent transparent c2fbbf8d-9b9e-4a0d-aca9-c188dc2b1a7e {} Transparent transparent f5dc263e-803f-482d-8246-5d81b262d7a6 {} Transparent transparent e2c48589-e004-4bb6-9183-018dc4053d79 {} Transparent transparent f31d1a78-41f0-4118-b2a2-59eef6ae3fc7 {} Transparent transparent 6645bbf2-c0cf-47f0-91d3-8d7edd74c2f3 {} Transparent transparent 04dc5700-6b85-44cc-b3b0-5444e2d8e26b {} Transparent transparent e0974078-8602-4170-9858-27edd7249918 {} Transparent transparent eaef6a0c-5ee0-4f4a-aee3-32a8ef39310f {} Transparent transparent 55998ed7-e204-4d69-aee3-7e2651c0ff56 {} Transparent

m-kostrzewa commented 7 years ago

Forgot to mention, I'm using Server 2016.

msabansal commented 7 years ago

@m-kostrzewa Ok it seems that this is happening because our logic dictates binding transparent networks to Network adapters with IPv4 addresses. When the network is deleted it takes some time for the reinitialized adapter to acquire IP which leads to the failure as we can't select the adapter till it has acquired the IP.

One of the workaround would be to specify the network adapter by specifying the name using the com.docker.network.windowsshim.interface option

m-kostrzewa commented 7 years ago 
PS > New-ContainerNetwork -Name net1 -Mode transparent -NetworkAdapterName Ethernet

Name Id                                   Subnets Mode        SourceMac DNSServers DNSSuffix
---- --                                   ------- ----        --------- ---------- ---------
net1 2f6e5b1f-dd62-4a2a-ab26-459452b82fb9 {}      Transparent

PS > Remove-ContainerNetwork -Name net1; New-ContainerNetwork -Name net2 -Mode transparent -NetworkAdapterName Ethernet

Confirm
Remove-ContainerNetwork will remove the container network "net1".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y
New-ContainerNetwork : Element not found.
At line:1 char:37
+ ... -Name net1; New-ContainerNetwork -Name net2 -Mode transparent -Networ ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

PS > Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet                  Realtek PCIe FE Family Controller            17 Up           34-17-EB-93-15-C9       100 Mbps
Wi-Fi                     Intel(R) Dual Band Wireless-AC 3160          13 Not Present  F4-06-69-1E-80-34          0 bps

Hmm.. Can't get it to work...

m-kostrzewa commented 7 years ago

I did a similar experiment:

PS C:\Users\mk> New-ContainerNetwork -name net -Mode Transparent -NetworkAdapterName Ethernet; Remove-ContainerNetwork -name net; 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -
NetworkAdapterName Ethernet; Get-NetAdapter | format-table; Start-Sleep -s 1; }

Confirm
Remove-ContainerNetwork will remove the container network "net".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"):
Name Id                                   Subnets Mode        SourceMac DNSServers DNSSuffix
---- --                                   ------- ----        --------- ---------- ---------
net  126aff1f-42c0-415b-8e13-defed91895eb {}      Transparent
New-ContainerNetwork : Element not found.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status       MacAddress        LinkSpeed
----                         --------------------              ------- ------       ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up           00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Disconnected 34-17-EB-93-15-C9     0 bps

New-ContainerNetwork : Element not found.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status       MacAddress        LinkSpeed
----                         --------------------              ------- ------       ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up           00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Disconnected 34-17-EB-93-15-C9     0 bps

New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : Unspecified error
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription              ifIndex Status MacAddress        LinkSpeed
----                         --------------------              ------- ------ ----------        ---------
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter       27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller       2 Up     34-17-EB-93-15-C9  100 Mbps

Name Id                                   Subnets Mode        SourceMac DNSServers DNSSuffix
---- --                                   ------- ----        --------- ---------- ---------
net  91fb8fff-5810-489f-8f58-242873bbdd97 {}      Transparent

Name                         InterfaceDescription                ifIndex Status MacAddress        LinkSpeed
----                         --------------------                ------- ------ ----------        ---------
vEthernet (HNSTransparent)   Hyper-V Virtual Ethernet Adapter #2      25 Up     34-17-EB-93-15-C9  100 Mbps
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter         27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller         2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : {Object Exists} An attempt was made to create an object and the object name already existed.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

Name                         InterfaceDescription                ifIndex Status MacAddress        LinkSpeed
----                         --------------------                ------- ------ ----------        ---------
vEthernet (HNSTransparent)   Hyper-V Virtual Ethernet Adapter #2      25 Up     34-17-EB-93-15-C9  100 Mbps
vEthernet (HNS Internal NIC) Hyper-V Virtual Ethernet Adapter         27 Up     00-15-5D-4B-78-77   10 Gbps
Ethernet                     Realtek PCIe FE Family Controller         2 Up     34-17-EB-93-15-C9  100 Mbps

New-ContainerNetwork : {Object Exists} An attempt was made to create an object and the object name already existed.
At line:1 char:127
+ ... 1..30 | % { New-ContainerNetwork -name net -Mode Transparent -Network ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [New-ContainerNetwork], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.Containers.PowerShell.Cmdlets.NewContainerNetwork

So it seems like you get "element not found" when Ethernet is in Disconnected state, and "unspecified error" when it's in Up state, but has no IP yet?

@msabansal, could you describe in a little bit more detail how to perform the workaround you mentioned? This doesn't seem to work:

PS C:\Users\mk> docker network rm $(docker network ls -q)
Error response from daemon: nat is a pre-defined network and cannot be removed
Error response from daemon: none is a pre-defined network and cannot be removed
PS C:\Users\mk> docker network create -d transparent -o com.docker.network.windowsshim.interface="Ethernet" net1; docker network rm net1; docker network create -d transparent -o com.docker.network.windowsshim.interface="Ethernet" net2;
2368224e9344b4d7dd7efe5dfb2cd446b2fc83d75ff6d746156ba23cad5d8cdc
net1
Error response from daemon: HNS failed with error : Element not found.
natalieparellano commented 5 years ago

We ran into a similar issue with a similar reproduction. We encountered this on Windows Server 1709 and 1803, but it looks to be resolved in 2019.