search

nokia / danm-utils

BSD 3-Clause "New" or "Revised" License
6 stars 9 forks source link

Policer: isolated Pods should be allowed some traffic by default even if their rules would normally forbid such connections #18

Closed Levovar closed 4 years ago

Levovar commented 4 years ago

When Policer isolates a Pod via the iptables provisioner the requested rules are created verbatim in its network namespace. However, some extra rules should be provisioned by default -regardless if operator requested them or not- to avoid uninentionally hindering "normal" communication. K8s applications usually depend on (cluster)DNS-based Services, so DNS traffic makes sense to be generally allowed, even if the DNS server's address isn't explicitly added as an Ingess/Egress target. In addition, if a Pod is an allowed Ingress/Egress target it makes sense to allow all related packets through the firewall for an already initiated and allowed transaction / connection. Real network communication is usually a two-way dialogue, i.e. if a Pod is allowed ICMP Egress we usually want to be able to receive the answer too :)

Levovar commented 4 years ago

Example ruleset after the change:

$ sudo nsenter -t 1316580 -n iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    9   804 DANM_INGRESS_V4  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    3   253 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
    6   551 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   11   897 DANM_EGRESS_V4  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    3   253 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 ctstate NEW,ESTABLISHED
    1    56 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 ctstate NEW,ESTABLISHED
    2   168 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

Chain DANM_EGRESS_V4 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    5   420 ACCEPT     all  --  *      *       0.0.0.0/0            10.240.1.100
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            10.244.0.125
    6   477 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DANM_INGRESS_V4 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       10.240.1.101         0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       10.244.2.78          0.0.0.0/0
    9   804 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Bi-directional ICMP works properly where we expect it to be, and is rejected otherwise:

$ sudo nsenter -t 1316580 -n ping 10.240.1.101
PING 10.240.1.101 (10.240.1.101) 56(84) bytes of data.
From 10.240.1.102 icmp_seq=1 Destination Port Unreachable
ping: sendmsg: Operation not permitted
From 10.240.1.102 icmp_seq=2 Destination Port Unreachable
ping: sendmsg: Operation not permitted
^C
--- 10.240.1.101 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1013ms

$ sudo nsenter -t 1316580 -n ping 10.240.1.100
PING 10.240.1.100 (10.240.1.100) 56(84) bytes of data.
64 bytes from 10.240.1.100: icmp_seq=1 ttl=64 time=0.284 ms
64 bytes from 10.240.1.100: icmp_seq=2 ttl=64 time=0.239 ms
64 bytes from 10.240.1.100: icmp_seq=3 ttl=64 time=0.197 ms
64 bytes from 10.240.1.100: icmp_seq=4 ttl=64 time=0.214 ms

DNS client traffic resolving Services also work as expected:

$ sudo nsenter -t 1337292 -n nslookup kubernetes.default.svc.nokia.net
Server:         172.31.3.154
Address:        172.31.3.154#53

Name:   kubernetes.default.svc.nokia.net
Address: 10.254.0.1