Guidance on using Zarf / k3s in RHEL 8.4, with a host-based firewall

[davidnraines]

When we deploy k3s, we are going to be required to block unneeded ports on the host that runs k3s. We’re running into a bit of trouble figuring out exactly how to configure what we need.

Some basic things we’ve concluded:

• k3s does not work with the Redhat “firewalld” service to be enabled
• As of RHEL 8, the legacy “iptables” system has been removed from Redhat
• This leaves “nftables” as the underlying Linux kernel subsystem that can filter packets (however, an executable “iptables” CLI still exists. It is an interface for the nftables subsystem)
• Through a mechanism we don’t fully understand, k3s appears to be generating its own nftables rulesets

We’ve written a set of nftables rules that we think should be compatible with the nftables configuration that k3s generates, but it never seems to quite work. One thing I’ve been struggling to understand is if we’re adding the wrong rules (if some other set of nft instructions in the attached .bash file would do the trick), or if there’s some other mismatch between us using nft and whatever k3s itself is doing.

Do you have any thoughts as to how we should go about blocking additional ports on a RHEL 8.4 host that is running k3s?

Attached is a script to add potential firewall rules: add_firewall_rules.txt

[RothAndrew]

@jeff-mccoy I recommended David come here so we could do a thread and get the rest of the team involved in troubleshooting this. This is outside my area of expertise so I need some help tracking down the issue.

[jeff-mccoy]

Copy @RothAndrew I'll dig into this some more, may need to phone a friend from the K3s folks.

[jeff-mccoy]

Back from Thanksgiving, can we setup a zoom or something this week to go through this @davidnraines and then post output back here after we solve it?

[davidnraines]

Based on our conversation yesterday, I have prepared additional documentation of what I have attempted This is relatively long, and we may want to Zoom again to discuss.

I have attached a ZIP file with a handful of artifacts to go along with the below description.

Observation:

• k3s is configuring its own IPTABLES rules
• k3s does this itself, and does not offer an obvious way to manipulate the IPTABLES rules
• You can see this happening in k3s source code, although I don’t fully understand what it is doing
• On Redhat Enterprise Linux 8, the “iptables” system has been removed.
• Instead RHEL uses a system called “nftables”.
• RHEL maps the iptables rules that k3s creates to nftables rules

My Questions:

• Are the rules that I’m trying to apply themselves incorrect, or am I trying to apply them incorrectly?
• Is there a recommended way to block additional ports on a RHEL 8 host that is running k3s?

What I have tried:

Scenario #1: I try to add nftables rules in a shell script

• In this scenario, we implement the nftables ruleset with the attached shell script (see k3s-networking.zip/scenario_1/add_firewall_rules.bash)
• The we use the “zarf init” command
• In this scenario, the pods never come online (See attached screenshot “scenario_1/pods_dont_deploy.png”)
• Additionally, there seem to be other networking issues (See attached screenshot “scenario_1/kubectl-cant-access-services.png”)

Scenario #2: I try to add nftables rules by the “/etc/sysconfig/nftables.conf” file

• In this scenario, I try to follow official RHEL documentation for using the nftables service
• https://www.redhat.com/en/blog/using-nftables-red-hat-enterprise-linux-8
• This may not be appropriate when using k3s

I ran the below commands

• ./zarf init --host localhost --components=management,utility-cluster –confirm
• systemctl enable nftables
• nft list ruleset > /etc/sysconfig/nftables.conf
• systemctl stop k3s
• reboot

My intention (had this worked) was to then modify the /etc/sysconfig/nftables.conf file to add additional port restrictions What I saw was that:

• The nftables service fails (see screenshot scenario_2/journalctl_nftables.png)
• It appears that the nftables service cannot parse the text form of the configuration that k3s generates

k3s-firewall-issues.zip .

[jeff-mccoy]

So I learned more than I wanted to about iptables/nftables/netfilter/etc. Tl;dr; use iptables in nftables mode not nftables directly. Example ruleset that worked with iptables-restore before launching k3s:

*filter
:INPUT ACCEPT [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -d 127.0.0.0/8 ! -i lo -j DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# SSH
-A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
# HTTP
-A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
# HTTPS
-A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
# Drop everything else
-A INPUT -i eth0 -j DROP
COMMIT

---

The issue lies here:

https://github.com/k3s-io/k3s/blob/master/pkg/agent/netpol/network_policy_controller.go#L299 https://github.com/coreos/go-iptables/blob/master/iptables/iptables.go#L223

Basically, under the hood it's just a fancy iptables execCommand which is 🤮. I played with a lot of iterations of setting via nft and even iptables-translate, but the root issue is even some basic nftables rulesets seems to break iptables in nftables mode. You can demonstrate this below:

1. Apply this ruleset (not the interface here is eth0)

   
   # drop any existing nftables ruleset
   flush ruleset

add table ip filter add chain ip filter INPUT { type filter hook input priority 0; policy accept; } add rule ip filter INPUT ct state related,established counter accept add rule ip filter INPUT iifname "eth0" ct state new tcp dport {22, 80, 443} counter accept add rule ip filter INPUT iifname "eth0" counter drop

2. Try to run `iptables -t filter -S INPUT` should produce the same error

<img width="1728" alt="Screen Shot 2021-12-15 at 1 10 28 AM" src="https://user-images.githubusercontent.com/882485/146164892-13f93b46-e63e-4559-a74b-2f624011aea1.png">

Let me know if this moves you in the right direction.

[jeff-mccoy]

This link is also really good for explaining the different variations of the firewall in RHEL now: https://www.redhat.com/en/blog/using-iptables-nft-hybrid-linux-firewall.

[jeff-mccoy]

I think this issue is resolved now, but we should update the docs with an example for someone who wants to do this @RothAndrew