search

projectcalico / k8s-exec-plugin

[Deprecated] Kubernetes Exec Plugin
Apache License 2.0
47 stars 12 forks source link

"WARNING Missing chain" DROP rule hit with calicoctl 0.14 #131

Closed fasaxc closed 8 years ago

fasaxc commented 8 years ago

Reported on Slack by user @slaw:

since I update calico to 0.14 and k8s plugin to 0.7.0, my pods cannot communicate to hosts outside the k8s cluster Tracing ping through iptables rules, it seems that I hit a DROP rule

DROP       all  --  anywhere             anywhere             /* WARNING Missing chain */

I asked them to submit a diags bundle.

slaws commented 8 years ago

Hello,

Some informations about my setup. I use a 172.23.0.0/16 pool with ipip enabled. The tests was made with a very simple RC and policy in annotations : 

{
  "kind": "ReplicationController",
  "apiVersion": "v1",
  "metadata": {
    "name": "baserc"
  },
  "spec": {
    "replicas": 3,
    "selector": {
      "application": "demo",
      "object": "rc",
      "parent": "none"
    },
    "template": {
        "metadata": {
          "labels": {
             "application": "demo",
             "object": "rc",
             "parent": "none"
          },
          "annotations": {
                "projectcalico.org/policy": "allow tcp from ports 80"
          }
        },
        "spec": {
          "containers": [
            {
                "name": "container1",
                "image": "nginx",
                "ports": [
                    {
                        "containerPort": 80,
                        "protocol": "TCP"
                    }
                ]
            }
          ]
        }
    }
  }
}

Pinging my DNS server from a pod of this RC : 

# kubectl exec -it  baserc-e2yyu /bin/bash
root@baserc-e2yyu:/# ping 192.168.128.1
PING 192.168.128.1 (192.168.128.1): 56 data bytes
^C--- 192.168.128.1 ping statistics ---
54 packets transmitted, 0 packets received, 100% packet loss

Tracing packets : 

Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392794] TRACE: raw:PREROUTING:policy:2 IN=cali8ee1b160bb8 OUT= MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392810] TRACE: nat:PREROUTING:rule:1 IN=cali8ee1b160bb8 OUT= MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392834] TRACE: nat:felix-PREROUTING:return:1 IN=cali8ee1b160bb8 OUT= MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392838] TRACE: nat:PREROUTING:rule:2 IN=cali8ee1b160bb8 OUT= MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392846] TRACE: nat:KUBE-SERVICES:return:29 IN=cali8ee1b160bb8 OUT= MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392849] TRACE: nat:PREROUTING:policy:3 IN=cali8ee1b160bb8 OUT= MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392857] TRACE: filter:FORWARD:rule:1 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392861] TRACE: filter:felix-FORWARD:rule:5 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392865] TRACE: filter:felix-FROM-ENDPOINT:rule:2 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392879] TRACE: filter:felix-FROM-EP-PFX-8:rule:3 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392911] TRACE: filter:felix-from-8ee1b160bb8:rule:1 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392928] TRACE: filter:felix-from-8ee1b160bb8:rule:3 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0 
Jan 15 14:35:09 kube-node-0 kernel: [ 2440.392963] TRACE: filter:felix-p-_4db76a0bb77b12e-o:rule:1 IN=cali8ee1b160bb8 OUT=eth0 MAC=4a:8d:6b:50:0f:01:ba:ec:8b:03:3b:43:08:00 SRC=172.23.0.17 DST=192.168.128.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=7363 DF PROTO=ICMP TYPE=8 CODE=0 ID=34 SEQ=0

Detail of the last chain : 

# iptables -L felix-p-_4db76a0bb77b12e-o
Chain felix-p-_4db76a0bb77b12e-o (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* WARNING Missing chain */

The full diags could be found at https://transfer.sh/zjl0M/diags-150116-141342.tar.gz

lxpollitt commented 8 years ago

cc/ @tomdee @caseydavenport

caseydavenport commented 8 years ago

Looks like Felix is failing to create the profile since the provided port is not an integer.

Relevant logs:

2016-01-15 13:54:57,237 [ERROR][63/1] calico.felix.fetcd 948: Validation failed for profile default_update-demo-kitten-dsx47_23d8b1a373eb rules: {'outbound_rules': [{'action': 'allow'}], 'id': u'default_update-demo-kitten-dsx47_23d8b1a373eb', 'inbound_rules': [{'action': 'allow', 'protocol': 'tcp', 'dst_ports': [u'80']}]}; ValidationFailed(u"Invalid port 80 (range unparseable) in rule {'action': 'allow', 'protocol': 'tcp', 'dst_ports': [u'80']}.",)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/calico/felix/fetcd.py", line 945, in parse_rules
    common.validate_rules(profile_id, rules)
  File "/usr/lib/python2.7/site-packages/calico/common.py", line 512, in validate_rules
    raise ValidationFailed(" ".join(issues))
ValidationFailed: Invalid port 80 (range unparseable) in rule {'action': 'allow', 'protocol': 'tcp', 'dst_ports': [u'80']}.

This PR https://github.com/projectcalico/libcalico/pull/53/files added validation in this area, but doesn't look like it asserts the given port is actually an integer (just that it can be cast as an int). I'd say this is a bug in the libcalico validation code, and probably a bug in the plugin for passing an int. Alternatively, maybe libcalico should just be smart enough to cast these as integers.

caseydavenport commented 8 years ago

@tomdee - I think we need to fix this for CNI 1.0.