ip-masq-agent conflict with cni rule

[xvdy]

k8s version: v1.13.2 host version: centos 7.3 behavor: ip-masq-agent not work, the ip in range of 10.0.0.0/8 does SNAT. Any thing wrong with my ip-masq-agent settings?

iptables:

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
CNI-1fbf321be92bf70398b61947  all  --  10.107.64.64/26      0.0.0.0/0            /* name: "cbr0" id: "8f8c0a517344e3ebf1e09399dda1d552b90a9474dcd5e30a4f598590e9c12ed5" */
CNI-e3cd2dddf7bb8b63f393b425  all  --  10.107.64.64/26      0.0.0.0/0            /* name: "cbr0" id: "55ff0c4ad643d52fb70be88585b3ab152f508b67350cf10b5880bfe306dc0274" */
IP-MASQ-AGENT  all  --  0.0.0.0/0            0.0.0.0/0            /* ip-masq-agent: ensure nat POSTROUTING directs all non-LOCAL destination traffic to our custom IP-MASQ-AGENT chain */ ADDRTYPE match dst-type !LOCAL

Chain CNI-1fbf321be92bf70398b61947 (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            10.107.64.64/26      /* name: "cbr0" id: "8f8c0a517344e3ebf1e09399dda1d552b90a9474dcd5e30a4f598590e9c12ed5" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "cbr0" id: "8f8c0a517344e3ebf1e09399dda1d552b90a9474dcd5e30a4f598590e9c12ed5" */

Chain CNI-2ee8d1d3c08248c9b227f9af (0 references)
target     prot opt source               destination

Chain CNI-30f721968232f9c23ba9544b (0 references)
target     prot opt source               destination

Chain CNI-6fe6f9122a41e4c9a2957bdc (0 references)
target     prot opt source               destination

Chain CNI-7bb5b1a56865087732f6db6a (0 references)
target     prot opt source               destination

Chain CNI-e3cd2dddf7bb8b63f393b425 (1 references)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            10.107.64.64/26      /* name: "cbr0" id: "55ff0c4ad643d52fb70be88585b3ab152f508b67350cf10b5880bfe306dc0274" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "cbr0" id: "55ff0c4ad643d52fb70be88585b3ab152f508b67350cf10b5880bfe306dc0274" */

Chain DOCKER (2 references)
target     prot opt source               destination

Chain IP-MASQ-AGENT (1 references)
target     prot opt source               destination
RETURN     all  --  0.0.0.0/0            169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  0.0.0.0/0            10.0.0.0/8           /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL

[MrHohn]

@xvdy Yep seems like the injected CNI rules capture majority of the traffic and masquerade them. What CNI plugin do you use?

cc @varunmar @grayluck

[BenTheElder]

any change you have "ipMasq": true on a bridge CNI? this looks like that, if so you should set "ipMasq": false in the CNI config instead when using the ip masq agent.

[xvdy]

Thank you for your help，I find the problem.It's my cni file.

{
  "name": "cbr0",
  "ipMasq": false, // missed set here which caused the problem
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "ipMasq": false,
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}