Review firewall rule as currently a pod can access https:// server outside of the VM

[cmoulliard]

Issues

A pod cannot access an external server and will report this error Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: i/o timeout.

Remark: We can curl this external http server within the VM

We should review/improve the firewall rules

[cmoulliard]

This stackoverflow post could certainly help us to figure out what to do : https://stackoverflow.com/questions/30449843/how-to-connect-a-kubernetes-pod-to-the-outside-world-without-a-forwarding-rule

[cmoulliard]

The log of the kube-proxy reports this error

ks logs kube-proxy-42kpq
W0226 11:18:38.884664       1 proxier.go:513] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0226 11:18:38.885941       1 proxier.go:513] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
...
I0226 11:18:39.019077       1 controller_utils.go:1036] Caches are synced for endpoints config controller
E0226 11:24:09.499368       1 proxier.go:706] Failed to ensure that filter chain KUBE-EXTERNAL-SERVICES exists: error creating chain "KUBE-EXTERNAL-SERVICES": exit status 3: modprobe: ERROR: could not insert 'ip_tables': Exec format error
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

REMARK: This message seems to appear when we stop the firewalld. But if we restart the FW and kill the pod, then it dont appear anymore

[snowdrop@n01-k115 ~]$ ks logs kube-proxy-l6nq2
W0226 14:12:01.201431       1 proxier.go:513] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0226 14:12:01.202786       1 proxier.go:513] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0226 14:12:01.203886       1 proxier.go:513] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0226 14:12:01.205036       1 proxier.go:513] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W0226 14:12:01.212059       1 server_others.go:249] Flag proxy-mode="" unknown, assuming iptables proxy
I0226 14:12:01.221301       1 server_others.go:143] Using iptables Proxier.
I0226 14:12:01.221651       1 server.go:534] Version: v1.15.9
I0226 14:12:01.232094       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0226 14:12:01.232124       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0226 14:12:01.232191       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0226 14:12:01.232238       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0226 14:12:01.232800       1 config.go:96] Starting endpoints config controller
I0226 14:12:01.232825       1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller
I0226 14:12:01.232842       1 config.go:187] Starting service config controller
I0226 14:12:01.232849       1 controller_utils.go:1029] Waiting for caches to sync for service config controller
I0226 14:12:01.333006       1 controller_utils.go:1036] Caches are synced for service config controller
I0226 14:12:01.333171       1 controller_utils.go:1036] Caches are synced for endpoints config controller

[cmoulliard]

Is the DNS server well defined as I get such error when I curl within the pod

[jboss@demo-ff5d9bc5b-45rjf ~]$ curl -k https://acme-v02.api.letsencrypt.org/directory
curl: (6) Could not resolve host: acme-v02.api.letsencrypt.org
[jboss@demo-ff5d9bc5b-45rjf ~]$ cat /etc/resolv.conf
nameserver 10.96.0.10
search demo.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

If I use the IP address, then we got an IP route issue

curl -k https://172.65.32.248/directory
curl: (7) Failed to connect to 172.65.32.248 port 443: No route to host

[cmoulliard]

If we stop the firewall, then that works

[jboss@demo-ff5d9bc5b-t955g ~]$ curl http://localhost:8080
{"timestamp":"2020-02-26T16:32:30.525+0000","path":"/","status":404,"error":"Not Found","message":null,"requestId":"2a96d8ad"}[jboss@demo-ff5d9bc5b-t955g ~]$
[jboss@demo-ff5d9bc5b-t955g ~]$ curl -k https://acme-v02.api.letsencrypt.org/directory
{
  "9VuaR79uG44": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}[jboss@demo-ff5d9bc5b-t955g ~]$

[cmoulliard]

For outgoing traffic towards http or https server, we need something like

iptables -A INPUT -p tcp -m multiport --dports 80,443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp -m multiport --sports 80,443 -m state --state ESTABLISHED -j ACCEPT

or this one to restrict to eth0 port

iptables -A OUTPUT -o eth0 -p tcp --dport 80,443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -i eth0 -p tcp --sport 80,443 -m state --state ESTABLISHED -j ACCEPT

WDYT @jacobdotcosta

[cmoulliard]

There is still an issue with latest change as I got the following error from the Cert Manager Challenge pod which contact Letsencrypt HTTP Server. So incoming traffic to the port 80 and eth0 IP is not allowed.

Waiting for http-01 challenge propagation: failed to perform self check
GET request 'http://k8s-console.94.130.110.67.nip.io/.well-known/acme-challenge/kYteZAmYx-Fxk3m-y6H8u35ZdcMb7hLMcOcQwAPnX6w':
Get http://k8s-console.94.130.110.67.nip.io/.well-known/acme-challenge/kYteZAmYx-Fxk3m-y6H8u35ZdcMb7hLMcOcQwAPnX6w:
dial tcp 94.130.110.67:80: connect: connection refused

@jacobdotcosta

[cmoulliard]

We can access the console https://k8s-console.94.130.110.67.nip.io/#/overview?namespace=default oken pwd shared on zulip

I added these rules and rebooted the VM

sudo iptables -A OUTPUT -o eth0 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -o eth0 -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -i eth0 -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT

To be validated of course @Antonio Costa

[jacobdotcosta]

Firewalld uses zones to manage traffic. With this in mind I configured the following zones, having as source a sample installation with podman I have in the office to test containers.

First I got the list of interfaces using nmcli.

# nmcli
xxxxxxxx: connected to System xxxxxxxx
...

xxxxxxxx, the physical interface, was added to the public zone. This zone is the default where ssh and other configurations where applied. This could be subject to discussion and move xxxxxxxx to the external zone.

# firewall-cmd --permanent --zone=public --add-interface=xxxxxxxx

To allow traffic related to k8s I added traffic having as source the k8s address range to the trusted zone.

# firewall-cmd --permanent --zone=trusted --add-source=xxx.xxx.xxx.xxx/xx

All sensitive data to be shared on zulip.

[cmoulliard]

the physical interface, was added to the public zone.

What are then the rules in/out applied to the interface assigned to a zone then ? @jacobdotcosta

[jacobdotcosta]

In the new installation I have to review the puiblic zone and try understanding why port 80 and 443 weren't open.

[cmoulliard]

In the new installation I have to review the puiblic zone and try understanding why port 80 and 443 weren't open.

This is not very well documented and the reason why I was questioning you ;-) Then, I think that it will be needed to also add additional rules to open the needed ports : 80, 443, ...

[cmoulliard]

FYI. Applying this rule was enough to allow the coreDNS pod to start and to access the Kube Api Service

sudo firewall-cmd --permanent --zone=trusted --add-source=10.244.0.0/16

During this step, the FW was modifying this file

/etc/firewalld/zones/trusted.xml
-->
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <short>Trusted</short>
  <description>All network connections are accepted.</description>
  <source address="10.244.0.0/16"/>
</zone>

No iptables commands were reported within the FW Log !

2020-03-03 18:51:59 DEBUG2: Introspect()
2020-03-03 18:51:59 DEBUG2: config.Introspect()
2020-03-03 18:51:59 DEBUG2: config.Introspect()
2020-03-03 18:51:59 DEBUG2: config.Introspect()
2020-03-03 18:51:59 DEBUG1: config.getZoneByName('trusted')
2020-03-03 18:51:59 DEBUG2: config.zone.7.Introspect()
2020-03-03 18:51:59 DEBUG1: config.zone.7.addSource('10.244.0.0/16')
2020-03-03 18:51:59 DEBUG1: config.zone.7.getSettings()
2020-03-03 18:51:59 DEBUG1: config.zone.7.update('...')
2020-03-03 18:51:59 DEBUG1: config.getZoneOfSource('10.244.0.0/16')
2020-03-03 18:51:59 DEBUG1: config.zone.7.Updated('trusted')
2020-03-03 18:52:04 DEBUG1: Loading zone file '/etc/firewalld/zones/trusted.xml'
2020-03-03 18:52:04 DEBUG1: config.zone.7.Updated('trusted')

It could be possible that the firewall debug parameter dont report the iptables modified ... as I can see the change

sudo iptables -S KUBE-FORWARD
-N KUBE-FORWARD
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod source rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "kubernetes forwarding conntrack pod destination rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Question: Do we need absolutely to use firewalld instead of iptables - see: https://www.digitalocean.com/community/tutorials/how-to-migrate-from-firewalld-to-iptables-on-centos-7 ?

@jacobdotcosta

[cmoulliard]

This rule is only needed sudo firewall-cmd --permanent --zone=trusted --add-source=10.244.0.0/16. We have tested it