Help troubleshooting

[feitnomore]

I'm trying to develop a tool to write some nftables rules for me. When I run it in one of my systems it works pretty fine, and when I move try to run it on another one, I get an error stating invalid argument, however I've got no details on what argument is failing:

nft.conn.Flush() failed: conn.Receive: netlink receive: invalid argument 

Is there any way to get more details on the error? The rule is exactly the same on both systems, but on one of them I get this error when flushing and on the other one not. Any clues or tips on how to troubleshoot will be greatly appreciated. Note: If I add the rule by hand on the system that is erroring out, it works fine.

Thanks!

[stapelberg]

Can you supply a minimal reproducer program for this issue please?

In general, the Linux kernel only returns EINVAL, not more details. You can check if dmesg contains any log lines.

Typically, if things work on one machine but not another, a different kernel version would explain that (maybe one machine is running a kernel that’s too old for the feature you’re trying to use).

[feitnomore]

func (nft *NFTables) buildIPSet(thisTable nftables.Table, sources []string) *nftables.Set {
    set := nftables.Set{
        Anonymous: true,
        Constant:  true,
        Name:      uuid.NewString()[0:7],
        ID:        rand.Uint32(), // rand.Intn(0xffff)),
        Table:     &thisTable,
        KeyType:   nftables.TypeIPAddr,
    }

    klog.Infof("IP ID: %d \n", set.ID)
    klog.Infof("IP Name: %s \n", set.Name)

    setElements := make([]nftables.SetElement, len(sources))
    for i := 0; i < len(sources); i++ {

        thisNet, thisIp, err := net.ParseCIDR(sources[i])
        if err != nil {
            klog.Errorf("ERROR: %v", err)
        }
        klog.Infof("thisNet: %v", thisNet)
        klog.Infof("thisIP: %v", thisIp)
        klog.Infof("thisMask: %v", net.IP(thisIp.Mask))
        thisMask := net.IP(thisIp.Mask).To4()
        if thisMask.Equal(net.ParseIP("255.255.255.255").To4()) {
            klog.Infof("SINGLE IP")
            klog.Infof("TEST: %v   ->  %s", thisIp.IP, thisIp.IP)
            setElements[i].Key = thisIp.IP
        } else {
            klog.Infof("MASK / NOT IMPLEMENTED YET")
            setElements[i].Key = thisIp.IP
        }
    }

    klog.Infof("elements: %v \n", setElements)
    if err := nft.conn.AddSet(&set, setElements); err != nil {
        klog.Errorf("ERROR: %v ", err)
    }

    klog.Infof("set: %v \n", set)
    if err := nft.conn.Flush(); err != nil {
        klog.Errorf("nft.conn.Flush() failed: %v \n", err)
    }

    return &set
}

This is the snippet in question. Here you have the output with errors:

I0903 18:19:28.725843       1 nft_tables.go:1100] IP ID: 542973396 
I0903 18:19:28.725850       1 nft_tables.go:1101] IP Name: a65849a 
I0903 18:19:28.725859       1 nft_tables.go:1110] thisNet: 172.16.145.3
I0903 18:19:28.725888       1 nft_tables.go:1111] thisIP: 172.16.145.3/32
I0903 18:19:28.725897       1 nft_tables.go:1112] thisMask: 255.255.255.255
I0903 18:19:28.725904       1 nft_tables.go:1115] SINGLE IP
I0903 18:19:28.725910       1 nft_tables.go:1116] TEST: 172.16.145.3   ->  172.16.145.3
I0903 18:19:28.725926       1 nft_tables.go:1124] elements: [{[172 16 145 3] [] [] false <nil> 0s 0s <nil>}] 
I0903 18:19:28.725962       1 nft_tables.go:1129] set: {0xc00054b6c0 542973396 a65849a true true false false false false false false 0s {ipv4_addr 4 7} { 0 0} <nil>} 
E0903 18:19:28.748946       1 nft_tables.go:1131] nft.conn.Flush() failed: conn.Receive: netlink receive: invalid argument 

And here you have an output from the working machine:

I0903 13:57:35.746930       1 nft_tables.go:1100] IP ID: 4286501312
I0903 13:57:35.746934       1 nft_tables.go:1101] IP Name: 850bd89
I0903 13:57:35.746939       1 nft_tables.go:1110] thisNet: 172.16.145.3
I0903 13:57:35.746945       1 nft_tables.go:1111] thisIP: 172.16.145.3/32
I0903 13:57:35.746955       1 nft_tables.go:1112] thisMask: 255.255.255.255
I0903 13:57:35.746961       1 nft_tables.go:1115] SINGLE IP
I0903 13:57:35.746966       1 nft_tables.go:1116] TEST: 172.16.145.3   ->  172.16.145.3
I0903 13:57:35.746972       1 nft_tables.go:1110] thisNet: 172.16.145.5
I0903 13:57:35.746976       1 nft_tables.go:1111] thisIP: 172.16.145.5/32
I0903 13:57:35.746981       1 nft_tables.go:1112] thisMask: 255.255.255.255
I0903 13:57:35.746986       1 nft_tables.go:1115] SINGLE IP
I0903 13:57:35.746990       1 nft_tables.go:1116] TEST: 172.16.145.5   ->  172.16.145.5
I0903 13:57:35.746995       1 nft_tables.go:1124] elements: [{[172 16 145 3] [] [] false <nil> 0s 0s <nil>} {[172 16 145 5] [] [] false <nil> 0s 0s <nil>}]
I0903 13:57:35.747021       1 nft_tables.go:1129] set: {0xc000835ee0 4286501312 850bd89 true true false false false false false false 0s {ipv4_addr 4 7} { 0 0} <nil>}

The rules are not exactly equal, but I've tried using exactly same values as well, and got the same behavior. Upon checking, there is a difference in kernel versions. The working one is 5.19.0-46-generic while the failing one is 5.15.0.119-generic.

Thanks

[feitnomore]

Got a new clue over here [ 3165.198775] nftables ruleset with unbound set, still trying to figure out.

[feitnomore]

Tried multiple things at once and solved, not sure what was, but basically: 1 - upgraded Kernel to 5.19.0-46-generic like the other machine 2 - loaded few kernel modules that were not loaded on the other box

Thanks @stapelberg for your support!