google / gvisor

Application Kernel for Containers
https://gvisor.dev
Apache License 2.0
15.63k stars 1.29k forks source link

failed to enable "PACKET_FANOUT" option: invalid argument #6124

Closed aojea closed 3 years ago

aojea commented 3 years ago

Description

I can't reliable reproduce it, but sometimes my code, that uses

    linkID, err := fdbased.New(&fdbased.Options{
        FDs:            []int{fd},
        MTU:            mtu,
        EthernetHeader: true,
        Address:        tcpip.LinkAddress(la),
        // Enable checksum generation as we need to generate valid
        // checksums for the veth device to deliver our packets to the
        // peer. But we do want to disable checksum verification as veth
        // devices do perform GRO and the linux host kernel may not
        // regenerate valid checksums after GRO.
        TXChecksumOffload:  false,
        RXChecksumOffload:  true,
        PacketDispatchMode: fdbased.RecvMMsg,
    })
    if err != nil {
        return fmt.Errorf("Can't create user-space link: %v\n", err)
    }

fails to create the link with the following error:

failed to enable "PACKET_FANOUT" option: invalid argument

The stack trace indicates the error comes from

https://github.com/google/gvisor/blob/f0b3298db07df63fa74559d5689008f9de9980a9/pkg/tcpip/link/fdbased/endpoint.go#L318-L324

After restarting the host, it works

Steps to reproduce

I have the suspicious that this happens when I create and delete a lot of raw sockets and bind them to gsvisor netstack in the same namespace (I mostly use the root namespace)

Environment

I use the netstack go module directly

        gvisor.dev/gvisor v0.0.0-20210506004418-fbfeba3024f0
kevinGC commented 3 years ago

I suspect this can be largely mitigated, but I don't know a way to avoid it completely besides running in a new network namespace.

This is likely a collision caused by using an existing fID. Since fanoutID always starts at 0 and increments by 1, creating multiple fdbased endpoints in the same network namespace should fail. This includes running multiple instances of netstack at the same time. A simple workaround is to use a value like the PID instead of 0 as a starting point, but as stated above namespaces are the best solution. I'll put this into a PR.