NICMx / Jool

SIIT and NAT64 for Linux
GNU General Public License v2.0
326 stars 66 forks source link

Add nftables support #285

Closed telmich closed 4 years ago

telmich commented 5 years ago

Currently jool does not work with nftables. However some networks switched away from ipXtables completely and thus jool cannot be used in those environments anymore.

I can be a beta tester, if there is a need for it.

ydahhrk commented 5 years ago

some networks switched away from ipXtables completely and thus jool cannot be used in those environments anymore.

Is Netfilter Jool useless as well? https://jool.mx/en/intro-jool.html#design

ydahhrk commented 4 years ago

I keep needing to reference the original emails when talking about this bug, so here are some quick links to them:

(question) https://mail-lists.nic.mx/pipermail/jool-list/2019-May/000247.html (response) https://mail-lists.nic.mx/pipermail/jool-list/2019-May/000248.html

telmich commented 4 years ago

If you need someone for testing -- just let me know!

ydahhrk commented 4 years ago

I have a functioning prototype, in case you want to hammer it.

Unfortunately, nft is not extensible, which means that I had to mess with its code instead of creating a plugin. For the moment, you need to compile a custom nft build to be able to use Jool's nftables wrapper. I will shortly interact with the Netfilter team and hopefully the situation will be improved.

Download, compile and install the custom nftables build (BTW: probably uninstall the old one beforehand):

(Edit 2020-04-28: Updated instructions to account for the new "jool" branches)

# I'm assuming you already have the autotools.
sudo apt install flex bison asciidoc libgmp-dev libreadline-dev libmnl-dev

git clone https://github.com/ydahhrk/libnftnl.git
cd libnftnl
git checkout jool
./autogen.sh
./configure
make
sudo make install
cd ..

git clone https://github.com/ydahhrk/nftables.git
cd nftables
git checkout jool
./autogen.sh
./configure
make
sudo make install
cd ..

sudo ldconfig

(I'm testing in Ubuntu 18.04.3, by the way.)

Download, compile and install the new Jool (BTW: probably uninstall the old one beforehand):

git clone https://github.com/NICMx/Jool.git
cd Jool
git checkout issue285
./autogen.sh
./configure
make
sudo make install
sudo dkms install .
# jool --version should print "4.0.7.2"

Basic SIIT recipe:

sudo modprobe jool_siit
sudo jool_siit instance add "nft-instance" --iptables -6 64:ff9b::/96

# the names of the table and chain are irrelevant, of course
sudo nft add table inet table1
sudo nft add chain inet table1 chain1 \{ type filter hook prerouting priority 0 \; \}
# first argument is the xlator type, second one is the instance name
sudo nft add rule  inet table1 chain1 jool siit "nft-instance"

Basic NAT64 recipe:

sudo modprobe jool
sudo jool instance add "nft-instance" --iptables -6 64:ff9b::/96

sudo nft add table inet table1
sudo nft add chain inet table1 chain1 \{ type filter hook prerouting priority 0 \; \}
sudo nft add rule  inet table1 chain1 jool nat64 "nft-instance"

That translates in my test environment.

Two notes:

  1. About --iptables: From Jool's point of view, nftables instances are literally the same as iptables instances, hence the flag. I'm considering deprecating --iptables in favor of something like "--tables" instead. Suggestions welcomed.
  2. I'm using the filter chain thingy, which is maybe wrong. I think the nat chain wouldn't work because it's defined to only accept the first packet of each stream, which is bad for Jool as currently implemented. But I haven't tested it.

How does it look?

ydahhrk commented 4 years ago

Here's the conversation in Netfilter Devel so far:

stv0g commented 4 years ago

Just for reference in case somebody else stumbles over this bug.

I am getting the following error message on CentOS 8 which is like caused because CentOS 8 switches to nftables:

Error: Error receiving the kernel module's response: Invalid input data or parameter

stv0g commented 4 years ago

@ydahhrk Your new nft support works like a charm 😁 👍

One idea to avoid patching nft: is it possible to incorporate the modification of the nft chains into the jool userspace binary?

ydahhrk commented 4 years ago

@ydahhrk Your new nft support works like a charm

@stv0g Thank you. With this bit of confirmation, and since it's pretty safe to say the mail thread was forgotten, I intend to upload a formal patch as another ping attempt tomorrow.

One idea to avoid patching nft: is it possible to incorporate the modification of the nft chains into the jool userspace binary?

Wouldn't it be problematic from a user's perspective? If nft doesn't understand Jool, this is the kind of stuff that happens:

$ # Prepare a table with a bunch of rules.
$ sudo nftables/src/nft add table inet graybox
$ sudo nftables/src/nft add chain inet graybox test \{ type filter hook prerouting priority 0 \; \}
$ sudo nftables/src/nft add rule inet graybox test ct state invalid drop
$ sudo nftables/src/nft add rule inet graybox test jool siit default
$ sudo nftables/src/nft add rule inet graybox test ct state invalid drop
$
$ # Print table, according to jool-aware nft
$ sudo nftables/src/nft list ruleset
table inet graybox {
    chain test {
        type filter hook prerouting priority filter; policy accept;
        ct state invalid drop
        jool "siit" "default"
        ct state invalid drop
    }
}
$
$ # Print table, according to jool-unaware nft
$ sudo nft list ruleset
table inet graybox {
    chain test {
        type filter hook prerouting priority filter; policy accept;
        ct state invalid drop

        ct state invalid drop
    }
}
netlink: Error: unknown expression type 'jool'.
stv0g commented 4 years ago

Oh yes. Thats a problem 👎

I had a brief look at the nftables-aware version of iptables. But I could't figure out if they support extensions.

ydahhrk commented 4 years ago

Poke sent:

I am getting the following error message on CentOS 8 which is like caused because CentOS 8 switches to nftables:

Error: Error receiving the kernel module's response: Invalid input data or parameter

What are you trying to do when this happens?

stv0g commented 4 years ago

What are you trying to do when this happens?

Adding an instance via jool instance

ydahhrk commented 4 years ago

Adding an instance via jool instance

Hmm... can you be more specific? I think this might be a bug. The error message is too generic, I can't find where it's being thrown, and I don't see how nft could be influencing it.

Can you still reproduce it? Might I have the versions of your kernel modules and userspace clients? Example:

$ sudo modprobe -r jool
$ sudo modprobe jool
$ dmesg -t | tail -1 # kernel module version
NAT64 Jool v4.0.8.0 module inserted.
$ jool --version # userspace client version
4.0.8.0

Also the arguments to the instance add command.

stv0g commented 4 years ago

@ydahhrk I cant reproduce my previous problem anymore. I suspect it was due to a version mismatch as you mentioned..

stv0g commented 4 years ago

I found a segfault in the netfilter code somewhere. It happens when I try to list the nft tables / chains:

(gdb) run list tables
Starting program: /usr/local/sbin/nft list tables
Missing separate debuginfos, use: yum debuginfo-install glibc-2.28-72.el8.x86_64
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
table ip nat
table ip filter
table ip mangle
table ip6 nat
table ip6 filter
table ip6 mangle
table ip6 raw
table ip raw
table inet jool-k8s-ziRE5S6H

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6eef133 in free () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install gmp-6.1.2-10.el8.x86_64 libmnl-1.0.4-6.el8.x86_64 ncurses-libs-6.1-7.20180224.el8.x86_64 readline-7.0-10.el8.x86_64
(gdb) bt
#0  0x00007ffff6eef133 in free () from /lib64/libc.so.6
#1  0x00007ffff7b60f6d in jool_stmt_destroy (stmt=0x74ccb0) at statement.c:677
#2  0x00007ffff7b62108 in stmt_free (stmt=0x74ccb0) at statement.c:51
#3  0x00007ffff7b621d8 in stmt_list_free (list=list@entry=0x619080) at statement.c:61
#4  0x00007ffff7b5cde0 in rule_free (rule=0x618eb0) at rule.c:631
#5  0x00007ffff7b5d79e in rule_free (rule=<optimized out>) at rule.c:864
#6  chain_free (chain=0x611010) at rule.c:864
#7  0x00007ffff7b5d876 in chain_free (chain=<optimized out>) at rule.c:1245
#8  table_free (table=0x607630) at rule.c:1245
#9  0x00007ffff7b5d9a1 in table_free (table=<optimized out>) at rule.c:1242
#10 __cache_flush (table_list=table_list@entry=0x605308) at rule.c:299
#11 0x00007ffff7b5d9cd in cache_release (cache=cache@entry=0x605300) at rule.c:305
#12 0x00007ffff7b86082 in nft_ctx_free (ctx=0x605260) at libnftables.c:294
#13 0x00000000004017f8 in main (argc=3, argv=0x7fffffffe598) at main.c:400
ydahhrk commented 4 years ago

@stv0g Try the following branches:

https://github.com/ydahhrk/libnftnl/tree/jool https://github.com/ydahhrk/nftables/tree/jool

stv0g commented 4 years ago

@ydahhrk Thanks the new branches fixed the segfault for me 👍

ydahhrk commented 4 years ago

Status update:

So the forecast is: My quick patch will not be accepted upstream.

In order to reconcile Jool and nftables, nothing less than a full merge will be required. This means this issue and #273 are one and the same.

Since this issue suddenly became a massive undertaking, my longterm plan is to finish #136 (because it's already in the testing phase), then implement #193, and then #285/#273.

(This excludes emerging urgent minor patches such as #326 and #325, which of course need to be addressed first.)

ydahhrk commented 4 years ago

Since this issue became a duplicate of #273, I will now close it.

In case anyone wants a quick 'n dirty version of this feature, my prototype is still available in my account and, as far as I know, is functional: