nftables support - Githubissues

niconorsk commented 6 years ago

Just wanted to put this up for other people like me who discover the inherent dependency. Is this something that is being considered at all? If not, any suggestions for people who want to use nftables to workaround this dependency.

pqarmitage commented 6 years ago

Yes, this is absolutely something I want to implement but I haven't got my head around nftables yet.

Any help would be much appreciated. For example, if you could provide the nftables commands to set up the equivalent functionality of what is set up using iptables/ipsets currently would be very helpful.

niconorsk commented 6 years ago

Posting info on behalf of my colleague who did the work:

Command line iptables calls are:

iptables -V to check whether iptables is available at all iptables -nL <chain> to check whether the user-configured chain for dropping exists iptables -A <chain> -d <virtual ip> -j DROP to drop packets for the virtual IP when in master mode if the virtual IP doesn't belong to the box iptables -D <chain> -d <virtual ip> -j DROP to remove the above when transitioning to backup For the first of these, iptables -V translates easily to nft -v. Unfortunately, things are less straightforward from there.

Unlike iptables, nftables doesn't come with a fixed set of tables or default chains. The actual hook points are mostly the same, but I think keepalived would need an extra config option for which table to add rules to, because it's not really possible to just assume that filter exists.

With this additional config, we can get:

iptables -nL <chain> -> nft list chain <table> <chain> iptables -A <chain> -d <virtual ip> -j DROP -> nft add rule <table> <chain> ip daddr 10.0.0.1 drop ...but then we get to rule deletion, which is much, much more annoying. Interactively, you do:

nft -a list chain <table> <chain>

which gives some output like:

table ip foo {
chain bar

{ ip daddr 10.0.0.1 drop # handle 33 }
}

and then to delete the rule:

nft delete rule foo bar handle 33

but I don't believe that you can get that handle without parsing output. Maybe libnftables has some API for this.

Clearly the rule deletion isn't good, so I guess the other option is maintaining some internal state of expected rules for each VRRP instance and making the user create a chain specifically for use by keepalived, so that it's easy to atomically flush and rewrite it when things change.

Also, the above is all for IPv4. For IPv6, the user would either need to have the table configured as inet, or (I think?) a duplicate ip6 table, and the add rule commands above would need to specify ip6 instead of `ip.

After investigating this, we realised that we don't actually want the blocking functionality that keepalived uses iptables for in our setup, but hope this helps anyways

pqarmitage commented 6 years ago

@niconorsk Many thanks for this update. What keepalived does currently is it adds entries to the chains specified in the vrrp_iptables option (it assumes that the iptables configuration has already been set up so that those chains are checked). keepalived will then use ipsets by default, rather than adding (potentially) long lists of addresses in an iptables chain. So, for example:

vrrp_iptables keepalived_in keepalived_out will cause keepalived to set up the following:

Chain keepalived_in (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set keepalived dst

Chain keepalived_out (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set keepalived src

and for IPv6:

Chain keepalived_in (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived_if6 dst,src ipv6-icmptype 135
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived_if6 dst,src ipv6-icmptype 136
    0     0 DROP       all      *      *       ::/0                 ::/0                 match-set keepalived_if6 dst,src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived6 dst ipv6-icmptype 135
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived6 dst ipv6-icmptype 136
    0     0 DROP       all      *      *       ::/0                 ::/0                 match-set keepalived6 dst

Chain keepalived_out (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived_if6 src,dst ipv6-icmptype 135
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived_if6 src,dst ipv6-icmptype 136
    0     0 DROP       all      *      *       ::/0                 ::/0                 match-set keepalived_if6 src,dst
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived6 src ipv6-icmptype 135
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 match-set keepalived6 src ipv6-icmptype 136
    0     0 DROP       all      *      *       ::/0                 ::/0                 match-set keepalived6 src

and the ipset configuration will look something like:

Name: keepalived
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 184
References: 2
Number of entries: 2
Members:
10.2.0.101
10.2.0.100

Name: keepalived6
Type: hash:ip
Revision: 4
Header: family inet6 hashsize 1024 maxelem 65536
Size in memory: 296
References: 6
Number of entries: 2
Members:
2001:470:69dd:2::4000
2001:470:69dd:2::3000

Name: keepalived_if6
Type: hash:net,iface
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536
Size in memory: 1328
References: 6
Number of entries: 1
Members:
fe80::4000,eth0

There is not a problem adding configuration to keepalived to specify the table(s) to use. It could be something like: vrrp_nftables TABLE:keepalived_in TABLE:keepalived_out

What would be really helpful would be to know the nft commands that would be needed (and I understand that TABLE will have to be specified), to set up the same configuration as I have listed above using iptables/ipsets.

niconorsk commented 6 years ago

Ok, so will post a bunch of nft commands below with the following assumptions:

Uses the above config with TABLE set to filter for both keepalived_in and keepalived_out chains
The surrounding nft infrastructure to check those chains is made in place
The provided chains should have no other rules(this makes deleting so much easier)
The code block can be saved to a file and run with nft -f or each line can be run individually by passing them as args to nft(need to escape semicolons in a shell though)

So here goes:

list chain inet filter keepalived_in
list chain inet filter keepalived_out
# Clear the chains
flush chain inet filter keepalived_in
flush chain inet filter keepalived_out
# Check set existence
list set inet filter keepalived
list set inet filter keepalived6
# Create the needed sets
create set inet filter keepalived { type ipv4_addr;}
create set inet filter keepalived6 { type ipv6_addr;}
# If needed flush sets
flush set  inet filter keepalived
flush set inet filter keepalived6
# Add elements to the sets
add element inet filter keepalived { 10.2.0.101,10.2.0.100 }
add element inet filter keepalived6 { 2001:470:69dd:2::4000, 2001:470:69dd:2::3000 }
# Add the appropriate DROP and accept rules
add rule inet filter keepalived_in ip daddr @keepalived drop
add rule inet filter keepalived_out ip daddr @keepalived drop
add rule inet filter keepalived_in icmpv6 type { nd-neighbor-solicit, nd-neighbor-advert } ip6 daddr @keepalived6 accept
add rule inet filter keepalived_in ip6 daddr @keepalived6 drop
add rule inet filter keepalived_out icmpv6 type { nd-neighbor-solicit, nd-neighbor-advert } ip6 daddr @keepalived6 accept
add rule inet filter keepalived_out ip6 daddr @keepalived6 drop
add rule inet filter keepalived_in icmpv6 type { nd-neighbor-solicit, nd-neighbor-advert }ip6 daddr { fe80::4000/64 }  iifname eth0 accept
add rule inet filter keepalived_in ip6 daddr { fe80::4000/64 }  iifname eth0 drop

At time of writing, nftables does not quite support doing exactly what you are doing with the keepalived_if6 set. Declaring it literally works but doing named concatenations does not yet work although it is a rapidly developed project so may change in the future.

Like was mentioned in the previous post, individual rule deletion is tricky which is why I went with the assumption of flushing the whole chain instead

pqarmitage commented 6 years ago

@niconorsk Many thanks for this information. I'm not sure what you mean by Declaring it literally works but doing named concatenations does not yet work though.

For your information, keepalived normally does not invoke the iptables/ip6tables/ipset commands, but uses functions in the ip4tc, ip6tc, xtables and ipset libraries. I would plan to make use of the library interface to nftables, but knowing what rules I need to create is really helpful. That might make rule deletion simpler too.

niconorsk commented 6 years ago

What I meant was this works: add rule inet filter keepalived_in ip6 daddr { fe80::4000/64 } iifname eth0 drop

But there is no support for doing simething like this:

create set inet filter keepalived_if6 { type ipv6_addr . ifname; flags interval;}
add element inet filter keepalived_if6 { fe80::4000/64 . eth0 }
add rule inet filter keepalived_in ip6 daddr . iifname @keepalived_if6 drop

pqarmitage commented 6 years ago

@niconorsk Many thanks for your guidance with this. When I have time I will implement nftables support.

acassen commented 6 years ago

I spent time around, I am a little sceptic and perplex about it. IMHO, if we 'just' need some basic filtering then best performances and future proof way would be TC filter and/or XDP (eBPF). I am right now digging into XDP and will provide a patch soon. Implementing TC filter or XDP will not require adding a new third party lib.

pqarmitage commented 5 years ago

Support for nftables now added. This gives an improvement over using iptables/ipsets. When XDP support is available, that will provide another option.

acassen / keepalived

nftables support #924