Allow both ingress and egress filtering on single interface

[pkolano]

Per our email discussion, please add the ability to simultaneously filter both ingress and egress on a single interface rather than requiring two interfaces. Since many systems only have one interface, this would allow zfw to be a more general-purpose firewall solution. It already seems like one of the most feature complete and production-ready eBPF firewalls out there so this would make it viable as a high-speed ip/nftables replacement for most purposes. Thanks for the hard work on this!

[r-caamano]

Hi. v0.8.3 should release sometime this week with both IPv4/IPv6 single interface ingress/egress filtering support. I will be testing over the next couple of days. This is the link to the candidate release branch if you are interested in previewing. https://github.com/netfoundry/zfw/tree/v0.8.3-release-candidate. The README describes basic setup and config.

[pkolano]

Awesome, thanks! I did try to preview, but seem to be getting errors that I didn't get with the last release I tried:

# /opt/openziti/bin/start_ebpf_router.py
ziti-router not installed, skipping ebpf router configuration!
Unable to retrieve LanIf!
ziti-router not installed, skipping ebpf router configuration!
Attempting to add ebpf egress to:  ens61f0
Ebpf not running no  maps to clear
tc parent add : ens61f0
libbpf: load bpf program failed: Permission denied
libbpf: -- BEGIN DUMP LOG ---

In the original release I tried, this worked (and still works with that version):

# /tmp/zfw.orig/bin/zfw -X ens61f0 -O /tmp/zfw.orig/bin/zfw_tc_ingress.o -z ingress
tc parent add : ens61f0
Set tc filter enable to 1 for ingress on ens61f0

In the new version, I get errors if I try the same thing:

# /opt/openziti/bin/zfw -X ens61f0 -O /opt/openziti/bin/zfw_tc_ingress.o -z ingress
tc parent add : ens61f0
libbpf: load bpf program failed: Permission denied
libbpf: -- BEGIN DUMP LOG ---

Don't know if maybe I messed something up copying the files over since the old version I used prebuilt files from the deb package while this time I built from source (also on a different system than where I deployed).

[r-caamano]

Hi. It's probably due to the compiling and moving to another system. We are close to completing our tests and will likely release v0.8.3 sometime tomorrow. After installing the new version and before running make sure you issue the sudo zfw -Q command or reboot as there were changes to an existing bpf map and you will get a conflict if not. If you have any issues with the released version please let me know.

[r-caamano]

v0.8.3 is now released!

[pkolano]

Thanks, the official release worked much better! Did a few tests and seems to be working nicely. I was wondering, is there any way to specify a block rule on the egress side for when you don't want to allow a connection somewhere? Right now, it seems to allow everything outbound, but maybe I missed something in the config/docs?

[r-caamano]

Hi @pkolano just want to make sure I understand the question.

If you set similar to the following, do you not see traffic blocked outbound on ens61f0?

sudo zfw -X ens61f0 -O /opt/openziti/bin/zfw_tc_ingress.o -z ingress
sudo zfw -X ens61f0 -O /opt/openziti/bin/zfw_tc_outbound_track.o -z egress
sudo zfw -b ens61f0

If you enter sudo zfw -L -E

You should see:

ens61f0: 
--------------------------
icmp echo               :0
verbose                 :0
ssh disable             :0
outbound_filter         :1
per interface           :0
tc ingress filter       :1
tc egress filter        :1
tun mode intercept      :0
vrrp enable             :0
eapol enable            :0
ddos filtering          :0
ipv6 enable             :0
--------------------------

The above should result in all outbound traffic except for arp and icmp to be dropped on ens61f0 (icmp echo-reply will also be dropped unless sudo zfw -e ens61f0 is set). ssh return traffic will also be allowed outbound unless ssh -x ens61f0 is set.

If you wanted to allow dns outbound to 8.8.8.8 you would add:

sudo zfw -I -c 8.8.8.8 -m 32 -l 53 -h 53 -t 0 -p udp -z egress

In order to survive reboot you would need to have fw-init.service enabled and have the following in /opt/openziti/etc/ebpf_config.json

{"InternalInterfaces":[], "ExternalInterfaces":[{"Name":"ens61f0", "PerInterfaceRules": false}]}

or equivalent InternalInterfaces config:

{"InternalInterfaces":[{"Name":"ens61f0", "OutboundPassThroughTrack": true}], "ExternalInterfaces":[]}

You would also need in /opt/openziti/bin/user/user_rules.sh

#!/bin/bash
sudo zfw -I -c 8.8.8.8 -m 32 -l 53 -h 53 -t 0 -p udp -z egress
sudo /opt/openziti/bin/zfw -b ens61f0

[r-caamano]

Hi @pkolano did the above guidance help or are you still unable to achieve the filtering you were expecting?

[r-caamano]

I have released a new version v0.8.5 which will not allow entering -b, --outbound-filtering <ifname> unless an egress filter exists on the interface which may have been the issue you ran into. It will also indicate the command to enter the egress tc filter in the usage message. My documentation on that in the README was not clear so I updated the README for the new version as well.

[r-caamano]

@pkolano v0.8.6 adds the ability to enter explicit deny rules by appending `-d, --disable to the -I, --insert rule to both ingress and egress rules. Rule precedence is based on longest match prefix. If the prefix is the same then the precedence follows the order entry of the rules, which when listed will go from top to bottom for ports with in the same prefixe.g.

If you wanted to allow all tcp 443 traffic outbound except to 10.1.0.0/16 you would enter the following egress rules:

sudo zfw -I -c 10.1.0.0 -m 16 -l 443 -h 443 -t 0 -p tcp -z egress -d
sudo zfw -I -c 0.0.0.0 -m 0 -l 443 -h 443 -t 0 -p tcp -z egress

Listing the above with sudo zfw -L -z egress you would see:

EGRESS FILTERS:
type   service id               proto   origin                  destination                     mapping:                                interface list                 
------ ----------------------   -----   -----------------   ------------------      ------------------------------------------------------- -----------------
accept 0000000000000000000000   tcp 0.0.0.0/0               0.0.0.0/0                       dpts=443:443        PASSTHRU to 0.0.0.0/0           []
deny   0000000000000000000000   tcp 0.0.0.0/0               10.1.0.0/16                     dpts=443:443        PASSTHRU to 10.1.0.0/16         []
Rule Count: 2 / 250000
prefix_tuple_count: 2 / 100000

The following illustrates the precedence with rules matching the same prefix:

Assume you want to block port 22 to address 172.16.240.137 and enter rules the following rules:

sudo zfw -I -c 172.16.240.139 -m 32 -l 1 -h 65535 -t 0 -p tcp -z egress
sudo zfw -I -c 172.16.240.139 -m 32 -l 22 -h 22 -t 0 -p tcp -z egress -d

sudo zfw -L -z egress
EGRESS FILTERS:
type   service id               proto   origin                  destination                     mapping:                                interface list                 
------ ----------------------   -----   -----------------   ------------------      ------------------------------------------------------- -----------------
accept 0000000000000000000000   tcp 0.0.0.0/0               172.16.240.139/32               dpts=1:65535        PASSTHRU to 172.16.240.139/32   []
deny   0000000000000000000000   tcp 0.0.0.0/0               172.16.240.139/32               dpts=22:22          PASSTHRU to 172.16.240.139/32   []
Rule Count: 2 / 250000

The rule listing shows the accept port range 1-65535 listed first and then the deny port 22 after. This would result in port 22 being allowed outbound because traffic would match the accept rule and never reach the deny rule.

The correct rule order entry would be:

sudo zfw -I -c 172.16.240.139 -m 32 -l 22 -h 22 -t 0 -p tcp -z egress -d
sudo zfw -I -c 172.16.240.139 -m 32 -l 1 -h 65535 -t 0 -p tcp -z egress

sudo zfw -L -z egress
EGRESS FILTERS:
type   service id               proto   origin                  destination                     mapping:                                interface list                 
------ ----------------------   -----   -----------------   ------------------      ------------------------------------------------------- -----------------
deny   0000000000000000000000   tcp 0.0.0.0/0               172.16.240.139/32               dpts=22:22          PASSTHRU to 172.16.240.139/32   []
accept 0000000000000000000000   tcp 0.0.0.0/0               172.16.240.139/32               dpts=1:65535        PASSTHRU to 172.16.240.139/32   []
Rule Count: 2 / 250000
prefix_tuple_count: 1 / 100000

This will result in traffic to port 22 matching the first rule and being dropped.

[r-caamano]

Hi @pkolano did the above feature additions meet your expectations?

[pkolano]

sorry, been busy this week. I will test it out next week. Thanks

[r-caamano]

Ok thanks @pkolano, no rush . Before testing please update to latest release which is v0.8.9 I added a bug fix for the -F, --flush operation.

[pkolano]

sorry, finally got back to this. Was trying to measure any performance degradation with randomly generated ip/port combos. When I was adding the rules, however, I am getting the error below (shown for one random ip/port that wasn't applied). I had added 2500 rules, which is shown in the tuple count, but doesn't seem to be applying them after a certain point. Any suggestions?

# zfw -I -c 55.119.51.158 -m 32 -l 47290 -h 47290 -t 0 -p tcp -z egress -d
Adding TCP mapping
MAP_UPDATE_ELEM: Argument list too long

# zfw --version
0.8.12

# zfw -L -z egress |tail -2
Rule Count: 102 / 250000
prefix_tuple_count: 2503 / 100000

[r-caamano]

Hi @pkolano I see what the issue is. Its actually a missing compilation argument issue. Thanks for catching this. When the package was compiled there should have been an argument -D MAX_BPF_ENTRIES=100000 which is missing right now in the workflow and Makefile specifically for building zfw_tc_outbound_track.o. Without this it defaults to only 100 egress map entries. I will be releasing v0.8.13 shortly which I will fix this. Sorry for this oversight.

[r-caamano]

@pkolano v0.8.13 is now released and should fix the MAP_UPDATE_ELEM: Argument list too long issue reported above.

[pkolano]

Looking really good now. No degradation of bandwidth or increase of latency/CPU utilization. Thanks so much for these additions. Is a very useful general purpose tool now and love the ease of installation/use compared to the other eBPF solutions I've looked at!

 Rules      Mbps    Latency      CPU
========    ====    =======     ====
Disabled    9500     0.043      10.5
    0       9500     0.045       9.9
  1000      9500     0.045      10.0
  2500      9500     0.045      11.5
  5000      9500     0.045      10.8
 10000      9500     0.042       9.5
100000      9500     0.042      10.2

[r-caamano]

Hi @pkolano thanks for taking the time to test out this project and for the great feedback and suggestions.