search

Azure / azure-container-networking

Azure Container Networking Solutions for Linux and Windows Containers
MIT License
377 stars 241 forks source link

Unable to access docker container application via CNI #634

Closed szikk closed 2 years ago

szikk commented 4 years ago

What happened:

I've setup an nginx server to run in a container with CNI. The container is accessible via ICMP but TCP traffic is now flowing towards the container ends up with Connection refused. Outbound traffic is also not available.

What you expected to happen:

Be able to access the container application via the VNET IP

How to reproduce it:

  1. Add static ipconfig to Host VM NIC
  2. sudo ./install-cni-plugin.sh v1.1.6 v0.8.6
  3. sudo ./docker-run.sh nginx default nginx
  4. curl 172.16.0.6 curl: (7) Failed to connect to 172.16.0.6 port 80: Connection refused

Orchestrator and Version (e.g. Kubernetes, Docker):

Docker version 19.03.12, build 48a66213fe

Operating System (Linux/Windows):
Linux

Kernel (e.g. uanme -a for Linux or $(Get-ItemProperty -Path "C:\windows\system32\hal.dll").VersionInfo.FileVersion for Windows): 18.04.1-Ubuntu SMP Mon Jul 13 12:54:45 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Anything else we need to know?:

cat /var/log/azure-vnet.log:

2020/07/29 16:04:06 [3210] Connected to telemetry service
2020/07/29 16:04:06 [3210] [cni-net] Plugin azure-vnet version v1.1.6.
2020/07/29 16:04:06 [3210] [cni-net] Running on Linux version 5.3.0-1034-azure (buildd@lcy01-amd64-020) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #35~18.04.1-Ubuntu SMP Mon Jul 13 12:54:45 UTC 2020
2020/07/29 16:04:06 [3210] [Azure-Utils] iptables --version
2020/07/29 16:04:06 [3210] [cni-net] iptable version:iptables v1.6.1, err:<nil>
2020/07/29 16:04:06 [3210] [Azure-Utils] ebtables --version
2020/07/29 16:04:06 [3210] [cni-net] ebtable version ebtables v2.0.10-4 (December 2011), err:<nil>
2020/07/29 16:04:06 [3210] [net] Network interface: {Index:1 MTU:65536 Name:lo HardwareAddr: Flags:up|loopback} with IP: [127.0.0.1/8 ::1/128]
2020/07/29 16:04:06 [3210] [net] Network interface: {Index:2 MTU:1500 Name:eth0 HardwareAddr:00:0d:3a:bc:57:da Flags:up|broadcast} with IP: [172.16.0.6/26 fe80::20d:3aff:febc:57da/64]
2020/07/29 16:04:06 [3210] [net] Network interface: {Index:3 MTU:1500 Name:br-3935bec577a7 HardwareAddr:02:42:f9:c3:a2:f0 Flags:up|broadcast|multicast} with IP: [172.18.0.1/16]
2020/07/29 16:04:06 [3210] [net] Network interface: {Index:4 MTU:1500 Name:docker0 HardwareAddr:02:42:bf:04:f3:52 Flags:up|broadcast|multicast} with IP: [172.17.0.1/16]
2020/07/29 16:04:06 [3210] [net] Network interface: {Index:5 MTU:1500 Name:azure0 HardwareAddr:00:0d:3a:bc:57:da Flags:up|broadcast|multicast} with IP: [172.16.0.6/26 172.16.0.4/26 fe80::20d:3aff:febc:57da/64]
2020/07/29 16:04:06 [3210] [net] reboot time 2020-07-29 15:35:28 +0000 UTC store mod time 2020-07-29 16:03:27.013557366 +0000 UTC
2020/07/29 16:04:06 [3210] [net] Restored state, &{Version:v1.1.6 TimeStamp:2020-07-29 16:03:27.01459973 +0000 UTC ExternalInterfaces:map[eth0:0xc0000e6f00] store:0xc000093230 Mutex:{state:1 sema:0}}
2020/07/29 16:04:06 [3210] External Interface &{Name:eth0 Networks:map[azure:0xc00016e270] Subnets:[172.16.0.0/26] BridgeName:azure0 DNSInfo:{Suffix:ovb3turwqniuxpapdx1nblletg.ax.internal.cloudapp.net Servers:[168.63.129.16] Options:[]} MacAddress:00:0d:3a:bc:57:da IPAddresses:[172.16.0.6/26 172.16.0.4/26] Routes:[0xc00014eab0] IPv4Gateway:172.16.0.1 IPv6Gateway:::}
2020/07/29 16:04:06 [3210] Number of endpoints: 0
2020/07/29 16:04:06 [3210] [cni-net] Plugin started.
2020/07/29 16:04:06 [3210] [cni-net] Processing ADD command with args {ContainerID:357419a70e440df151a25f210704bf543c33557fd5401d906b728087dc8260b8 Netns:/proc/3150/ns/net IfName:eth0 Args:K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx Path:/opt/cni/bin StdinData:{ "type": "azure-vnet", "mode": "bridge", "bridge": "azure0", "ipsToRouteViaHost": [ "169.254.20.10" ], "ipam": { "type": "azure-vnet-ipam" }, "name": "azure", "cniVersion": "0.3.0" }
}.
2020/07/29 16:04:06 [3210] [cni-net] Read network configuration &{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: InfraVnetAddressSpace: IPV6Mode: ServiceCidrs: VnetCidrs: PodNamespaceForDualNetwork:[] IPsToRouteViaHost:[169.254.20.10] MultiTenancy:false EnableSnatOnHost:false EnableExactMatchForPodName:false DisableHairpinOnHostInterface:false DisableIPTableLock:false CNSUrl: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet: Address: QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.
2020/07/29 16:04:06 [3210] Get number of endpoints for ifname eth0 network azure
2020/07/29 16:04:06 [3210] Result from multitenancy <nil>
2020/07/29 16:04:06 [3210] Trying to retrieve endpoint id 357419a7-eth0
2020/07/29 16:04:06 [3210] [cni-net] Found network azure with subnet 172.16.0.0/26.
2020/07/29 16:04:06 [3210] [cni] Calling plugin azure-vnet-ipam ADD nwCfg:&{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: InfraVnetAddressSpace: IPV6Mode: ServiceCidrs: VnetCidrs: PodNamespaceForDualNetwork:[] IPsToRouteViaHost:[169.254.20.10] MultiTenancy:false EnableSnatOnHost:false EnableExactMatchForPodName:false DisableHairpinOnHostInterface:false DisableIPTableLock:false CNSUrl: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet:172.16.0.0/26 Address: QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.
2020/07/29 16:04:06 [3210] [cni] Plugin azure-vnet-ipam returned result:IP:[{Version:4 Interface:<nil> Address:{IP:172.16.0.6 Mask:ffffffc0} Gateway:172.16.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:172.16.0.1}], DNS:{Nameservers:[168.63.129.16] Domain: Search:[] Options:[]}, err:<nil>.
2020/07/29 16:04:06 [3210] [cni-net] Creating endpoint 357419a7-eth0.
2020/07/29 16:04:06 [3210] [net] Creating endpoint &{Id:357419a7-eth0 ContainerID:357419a70e440df151a25f210704bf543c33557fd5401d906b728087dc8260b8 NetNsPath:/proc/3150/ns/net IfName:eth0 SandboxKey: IfIndex:0 MacAddress: DNS:{Suffix: Servers:[168.63.129.16] Options:[]} IPAddresses:[{IP:172.16.0.6 Mask:ffffffc0}] IPsToRouteViaHost:[169.254.20.10] InfraVnetIP:{IP:<nil> Mask:<nil>} Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} Src:<nil> Gw:172.16.0.1 Protocol:0 DevName: Scope:0 Priority:0}] Policies:[] Gateways:[] EnableSnatOnHost:false EnableInfraVnet:false EnableMultiTenancy:false EnableSnatForDns:false AllowInboundFromHostToNC:false AllowInboundFromNCToHost:false NetworkContainerID: PODName:nginx PODNameSpace:default Data:map[vethname:azure357419a70e440df151a25f210704bf543c33557fd5401d906b728087dc8260b8eth0] InfraVnetAddressSpace: SkipHotAttachEp:false IPV6Mode: VnetCidrs: ServiceCidrs:} in network azure.
2020/07/29 16:04:06 [3210] Generate veth name based on the key provided azure357419a70e440df151a25f210704bf543c33557fd5401d906b728087dc8260b8eth0
2020/07/29 16:04:06 [3210] Bridge client
2020/07/29 16:04:06 [3210] [net] Creating veth pair azv20be1f01e0f azv20be1f01e0f2.
2020/07/29 16:04:06 [3210] [net] Setting link azv20be1f01e0f state up.
2020/07/29 16:04:06 [3210] [Azure-Utils] sysctl -w net.ipv6.conf.azv20be1f01e0f.accept_ra=0
2020/07/29 16:04:06 [3210] [net] Setting link azv20be1f01e0f master azure0.
2020/07/29 16:04:06 [3210] [net] Adding ARP reply rule for IP address 172.16.0.6/26
2020/07/29 16:04:06 [3210] [Azure-Utils] ebtables -t nat -A PREROUTING -p ARP --arp-op Request --arp-ip-dst 172.16.0.6 -j arpreply --arpreply-mac 22:83:66:2b:7e:11 --arpreply-target DROP
2020/07/29 16:04:06 [3210] [net] Adding MAC DNAT rule for IP address 172.16.0.6/26
2020/07/29 16:04:06 [3210] [Azure-Utils] ebtables -t nat -A PREROUTING -p IPv4 -i eth0 --ip-dst 172.16.0.6 -j dnat --to-dst 22:83:66:2b:7e:11 --dnat-target ACCEPT
2020/07/29 16:04:06 [3210] [net] Adding static arp for IP address 172.16.0.6/26 and MAC 22:83:66:2b:7e:11 in VM
2020/07/29 16:04:06 [3210] [net] Checking if EB rule -p IPv4 --ip-dst 169.254.20.10 -j redirect already exists in table broute chain BROUTING
2020/07/29 16:04:06 [3210] [Azure-Utils] ebtables -t broute -L BROUTING --Lmac2
2020/07/29 16:04:06 [3210] [net] EB rule -p IPv4 --ip-dst 169.254.20.10 -j redirect already exists in table broute chain BROUTING.
2020/07/29 16:04:06 [3210] [net] Setting hairpin for hostveth azv20be1f01e0f
2020/07/29 16:04:06 [3210] [net] Opening netns /proc/3150/ns/net.
2020/07/29 16:04:06 [3210] [net] Setting link azv20be1f01e0f2 netns /proc/3150/ns/net.
2020/07/29 16:04:06 [3210] [net] Entering netns /proc/3150/ns/net.
2020/07/29 16:04:06 [3210] [net] Setting link azv20be1f01e0f2 state down.
2020/07/29 16:04:06 [3210] [net] Setting link azv20be1f01e0f2 name eth0.
2020/07/29 16:04:06 [3210] [Azure-Utils] sysctl -w net.ipv6.conf.eth0.accept_ra=0
2020/07/29 16:04:06 [3210] [net] Setting link eth0 state up.
2020/07/29 16:04:06 [3210] [net] Adding IP address 172.16.0.6/26 to link eth0.
2020/07/29 16:04:06 [3210] [net] Adding IP route {Dst:{IP:0.0.0.0 Mask:00000000} Src:<nil> Gw:172.16.0.1 Protocol:0 DevName: Scope:0 Priority:0} to link eth0.
2020/07/29 16:04:06 [3210] [net] Exiting netns /proc/3150/ns/net.
2020/07/29 16:04:06 [3210] [net] Created endpoint &{Id:357419a7-eth0 HnsId: SandboxKey: IfName:azv20be1f01e0f2 HostIfName:azv20be1f01e0f MacAddress:22:83:66:2b:7e:11 InfraVnetIP:{IP:<nil> Mask:<nil>} LocalIP: IPAddresses:[{IP:172.16.0.6 Mask:ffffffc0}] Gateways:[172.16.0.1] DNS:{Suffix: Servers:[168.63.129.16] Options:[]} Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} Src:<nil> Gw:172.16.0.1 Protocol:0 DevName: Scope:0 Priority:0}] VlanID:0 EnableSnatOnHost:false EnableInfraVnet:false EnableMultitenancy:false AllowInboundFromHostToNC:false AllowInboundFromNCToHost:false NetworkContainerID: NetworkNameSpace:/proc/3150/ns/net ContainerID:357419a70e440df151a25f210704bf543c33557fd5401d906b728087dc8260b8 PODName:nginx PODNameSpace:default InfraVnetAddressSpace: NetNs:}.
2020/07/29 16:04:06 [3210] [net] Save succeeded.
2020/07/29 16:04:06 [3210] Get number of endpoints for ifname eth0 network azure
2020/07/29 16:04:06 [3210] [cni-net] ADD command completed with result:Interfaces:[{Name:eth0 Mac: Sandbox:}], IP:[{Version:4 Interface:<nil> Address:{IP:172.16.0.6 Mask:ffffffc0} Gateway:172.16.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:172.16.0.1}], DNS:{Nameservers:[168.63.129.16] Domain: Search:[] Options:[]} err:<nil>.
2020/07/29 16:04:06 [3210] [cni-net] Plugin stopped.
2020/07/29 16:04:06 [3210] Sending report succeeded

cat /var/log/azure-vnet-ipam.log:


2020/07/29 16:03:27 [3057] [cni-ipam] Plugin azure-vnet-ipam version v1.1.6.
2020/07/29 16:03:27 [3057] [cni-ipam] Running on Linux version 5.3.0-1034-azure (buildd@lcy01-amd64-020) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #35~18.04.1-Ubuntu SMP Mon Jul 13 12:54:45 UTC 2020
2020/07/29 16:03:27 [3057] [ipam] reboot time 2020-07-29 15:35:28 +0000 UTC store mod time 2020-07-29 15:51:58.07650337 +0000 UTC
2020/07/29 16:03:27 [3057] [ipam] Restored state, &{Version:v1.1.6 TimeStamp:2020-07-29 15:51:58.07878444 +0000 UTC AddrSpaces:map[local:0xc00018c4b0] store:0xc0002b2bd0 source:<nil> netApi:<nil> Mutex:{state:0 sema:0}}
2020/07/29 16:03:27 [3057] [cni-ipam] Plugin started.
2020/07/29 16:03:27 [3057] [cni-ipam] Processing DEL command with args {ContainerID:50db5922fada75ffba532e36988dfab7b777bdad1879233257cb7cd8a7f741df Netns:netnspath IfName:eth0 Args:K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx2 Path:/opt/cni/bin StdinData:{"cniVersion":"0.3.0","name":"azure","type":"azure-vnet","mode":"bridge","master":"","bridge":"azure0","ipsToRouteViaHost":["169.254.20.10"],"Ipam":{"type":"azure-vnet-ipam","subnet":"172.16.0.0/26","ipAddress":"172.16.0.6"},"dns":{},"runtimeConfig":{"dns":{}},"AdditionalArgs":null}}.
2020/07/29 16:03:27 [3057] [cni-ipam] Read network configuration &{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: InfraVnetAddressSpace: IPV6Mode: ServiceCidrs: VnetCidrs: PodNamespaceForDualNetwork:[] IPsToRouteViaHost:[169.254.20.10] MultiTenancy:false EnableSnatOnHost:false EnableExactMatchForPodName:false DisableHairpinOnHostInterface:false DisableIPTableLock:false CNSUrl: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet:172.16.0.0/26 Address:172.16.0.6 QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.
2020/07/29 16:03:27 [3057] [ipam] Starting source azure.
2020/07/29 16:03:27 [3057] [ipam] Refreshing address source.
2020/07/29 16:03:27 [3057] [Utils] Initializing HTTP client with connection timeout: 10, response header timeout: 10
2020/07/29 16:03:27 [3057] [ipam] Wireserver call http://168.63.129.16/machine/plugins?comp=nmagent&type=getinterfaceinfov1 to retrieve IP List
2020/07/29 16:03:27 [3057] [ipam] Save succeeded.
2020/07/29 16:03:27 [3057] [ipam] Releasing address with address:172.16.0.6 options:map[].
2020/07/29 16:03:27 [3057] [ipam] Address release completed with address:172.16.0.6 err:<nil>.
2020/07/29 16:03:27 [3057] [ipam] Save succeeded.
2020/07/29 16:03:27 [3057] [cni-ipam] DEL command completed with err:<nil>.
2020/07/29 16:03:27 [3057] [cni-ipam] Plugin stopped.
2020/07/29 16:04:06 [3223] [cni-ipam] Plugin azure-vnet-ipam version v1.1.6.
2020/07/29 16:04:06 [3223] [cni-ipam] Running on Linux version 5.3.0-1034-azure (buildd@lcy01-amd64-020) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #35~18.04.1-Ubuntu SMP Mon Jul 13 12:54:45 UTC 2020
2020/07/29 16:04:06 [3223] [ipam] reboot time 2020-07-29 15:35:28 +0000 UTC store mod time 2020-07-29 16:03:27.0335586 +0000 UTC
2020/07/29 16:04:06 [3223] [ipam] Restored state, &{Version:v1.1.6 TimeStamp:2020-07-29 16:03:27.037790661 +0000 UTC AddrSpaces:map[local:0xc0002bd1d0] store:0xc0002bcbd0 source:<nil> netApi:<nil> Mutex:{state:0 sema:0}}
2020/07/29 16:04:06 [3223] [cni-ipam] Plugin started.
2020/07/29 16:04:06 [3223] [cni-ipam] Processing ADD command with args {ContainerID:357419a70e440df151a25f210704bf543c33557fd5401d906b728087dc8260b8 Netns:/proc/3150/ns/net IfName:eth0 Args:K8S_POD_NAMESPACE=default;K8S_POD_NAME=nginx Path:/opt/cni/bin StdinData:{"cniVersion":"0.3.0","name":"azure","type":"azure-vnet","mode":"bridge","master":"","bridge":"azure0","ipsToRouteViaHost":["169.254.20.10"],"Ipam":{"type":"azure-vnet-ipam","subnet":"172.16.0.0/26"},"dns":{},"runtimeConfig":{"dns":{}},"AdditionalArgs":null}}.
2020/07/29 16:04:06 [3223] [cni-ipam] Read network configuration &{CNIVersion:0.3.0 Name:azure Type:azure-vnet Mode:bridge Master: Bridge:azure0 LogLevel: LogTarget: InfraVnetAddressSpace: IPV6Mode: ServiceCidrs: VnetCidrs: PodNamespaceForDualNetwork:[] IPsToRouteViaHost:[169.254.20.10] MultiTenancy:false EnableSnatOnHost:false EnableExactMatchForPodName:false DisableHairpinOnHostInterface:false DisableIPTableLock:false CNSUrl: Ipam:{Type:azure-vnet-ipam Environment: AddrSpace: Subnet:172.16.0.0/26 Address: QueryInterval:} DNS:{Nameservers:[] Domain: Search:[] Options:[]} RuntimeConfig:{PortMappings:[] DNS:{Servers:[] Searches:[] Options:[]}} AdditionalArgs:[]}.
2020/07/29 16:04:06 [3223] [ipam] Starting source azure.
2020/07/29 16:04:06 [3223] [ipam] Refreshing address source.
2020/07/29 16:04:06 [3223] [Utils] Initializing HTTP client with connection timeout: 10, response header timeout: 10
2020/07/29 16:04:06 [3223] [ipam] Wireserver call http://168.63.129.16/machine/plugins?comp=nmagent&type=getinterfaceinfov1 to retrieve IP List
2020/07/29 16:04:06 [3223] [ipam] Save succeeded.
2020/07/29 16:04:06 [3223] [ipam] Requesting address with address: options:map[].
2020/07/29 16:04:06 [3223] [ipam] Address request completed with address:172.16.0.6/26 err:<nil>.
2020/07/29 16:04:06 [3223] [ipam] Save succeeded.
2020/07/29 16:04:06 [3223] [cni-ipam] Allocated address 172.16.0.6/26.
2020/07/29 16:04:06 [3223] [cni-ipam] ADD command completed with result:IP:[{Version:4 Interface:<nil> Address:{IP:172.16.0.6 Mask:ffffffc0} Gateway:172.16.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:172.16.0.1}], DNS:{Nameservers:[168.63.129.16] Domain: Search:[] Options:[]} err:<nil>.
2020/07/29 16:04:06 [3223] [cni-ipam] Plugin stopped.
matmerr commented 4 years ago

If this is a non aks/aks-engine enviroment, you may need to add a postrouting rule to the vm

https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md#using-cni-in-non-aks-environment-linux

szikk commented 4 years ago

Hello @matmerr the route was already added but I still did not have internet access from container and also the host vm did not have any connectivity to the container but other VM from inside vnet had. (I also disabled all the NSGs from vnet)

I might be getting a little bit confused here and I think is better to tell you the end goal. Basically I am trying to create Docker containers withing Azure VNET and with a Standard SLB balance traffic between them.

I tried to follow the instruction from here: https://docs.microsoft.com/en-us/azure/virtual-network/deploy-container-networking#deploy-plug-in-for-docker-containers

I also tried this approach: https://github.com/Azure/azure-container-networking/blob/master/docs/cnm.md but I ended up in the same place.

What would be the correct approach for this?

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days

github-actions[bot] commented 2 years ago

Issue closed due to inactivity.