Wrong source IP in pod-to-pod communication (kubenet)

[mwieczorek]

Is this a request for help?: no

Is this an ISSUE or FEATURE REQUEST? (choose one): issue

What version of acs-engine?: Version: v0.26.2

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm) Kubernetes

What happened: I run: acs-engine deploy -l westeurope -g example --dns-prefix example --api-model ./example.json

api-model:

{
    "apiVersion": "vlabs",
    "properties": {
      "orchestratorProfile": {
        "orchestratorType": "Kubernetes",
        "orchestratorRelease": "1.10",
        "kubernetesConfig": {
            "useManagedIdentity": true,
            "networkPlugin": "kubenet"
        }
      },
      "masterProfile": {
        "count": 1,
        "dnsPrefix": "",
        "vmSize": "Standard_D2_v2"
      },
      "agentPoolProfiles": [
        {
          "name": "agentpool1",
          "count": 3,
          "vmSize": "Standard_D2_v2",
          "availabilityProfile": "AvailabilitySet"
        }
      ],
      "linuxProfile": {
        "adminUsername": "azureuser",
        "ssh": {
            "publicKeys": [
                {
                    "keyData": "xxx"
                }
            ]
        }
      }
    }
  }

When the cluster is ready I run:

kubectl run source-ip-app --image=gcr.io/google_containers/echoserver:1.4
kubectl expose deployment source-ip-app --name=nodeport --port=80 --target-port=8080 --type=NodePort
kubectl run -it test --image=tutum/curl

Inside the pod I run: curl http://nodeport

And get response: client_address=10.244.3.1

10.244.3.1 is an IP of cbr0 bridge interface of VM where source-ip-app app is run

What you expected to happen:

I expect that I get f.e. client_address=10.244.3.6 (10.244.3.6 as a curl pod IP) not cbr0 IP.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know: It was running as I expect in acs-engine v0.22.4 (I didn't check versions between)

[mwieczorek]

I dig a little bit more. When I install with CNI=azure there's no such issue. I looked at the iptables: iptables -t nat -L POSTROUTING

When kubenet:

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        anywhere
MASQUERADE  all  --  anywhere            !0.0.0.0              /* kubenet: SNAT for outbound traffic from cluster */ ADDRTYPE match dst-type !LOCAL
IP-MASQ-AGENT  all  --  anywhere             anywhere             /* ip-masq-agent: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom IP-MASQ-AGENT chain */ ADDRTYPE match dst-type !LOCAL

When azure CNI:

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  anywhere             anywhere             /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        anywhere
IP-MASQ-AGENT  all  --  anywhere             anywhere             /* ip-masq-agent: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom IP-MASQ-AGENT chain */ ADDRTYPE match dst-type !LOCAL

So with kubenet there's additional rule:

MASQUERADE  all  --  anywhere            !0.0.0.0              /* kubenet: SNAT for outbound traffic from cluster */ ADDRTYPE match dst-type !LOCAL

When I delete the rule manually everything begins to work as I expect.

Is it an k8s/kubenet upstream issue? Or is it an issue in configuration of kubelet/ip-masq-agent?

[mwieczorek]

Ok, I found, iptables -t nat -L POSTROUTING shows that dest is 0.0.0.0, but when I used iptables-save I found out it's 0.0.0.0/32... When I switched to 0.0.0.0/0 it's also working...

So it looks like bad config of kubelet made by acs-engine. I'll prepare a fix