net/tstun: failed to set MTU of TUN device: invalid argument

[bradfitz]

[ copying from https://www.reddit.com/r/Tailscale/comments/1cegdm2/error_with_the_static_arm_build_1582/ ]

I use the Static ARM build of tailscale on a Seagate GoFlex Home NAS Device, which is running Debian Linux.

Up until version 1.58.2 everything has been working flawlessly.. however on any version above that I get an error when trying too bring up tailscale.

Linux kernel version: 5.13.6-kirkwood-tld-1
'modprobe tun' successful
/dev/net/tun: Dcrw-rw-rw-
wgengine.NewUserspaceEngine(tun "tailscale0") error: tstun.New("tailscale0"): failed to set MTU of TUN device: invalid argument
flushing log.
logger closing down
getLocalBackend error: createEngine: tstun.New("tailscale0"): failed to set MTU of TUN device: invalid argument

Reverting (copying the static binaries from 1.58.2 back over top of the newer ones).. fixes the issue every time.

Can anyone advise how I can get more detail about whats going wrong here?

[sambartle]

Kernel Config attached. config-5.13.6-kirkwood-tld-1.txt (I Realised I had a tarball of the rootfs on my machine)

[bradfitz]

Any chance you could git bisect and find the commit where it fails to start?

[sambartle]

Sure, I'll have a look tomorrow morning (I'm in the UK) and figure it out.

[bradfitz]

I looked at the diff from 1.58.2 to 1.60.0. Not a ton in there. Go 1.22 instead of Go 1.21, and bumps from golang.org/x/sys 0.15 to 0.16. One of those seems the most likely candidate.

Building 1.58.2 but with Go 1.22 and/or golang.org/x/sys 0.16 would be interesting.

Another thing that would be interesting would be strace -f tailscaled output to see the exact system calls that are failing .

[sambartle]

Checking out 1.58.2 and modifying it to build with Go 1.22.0 causes the same error once built (It works when built as is with 1.21 as a sanity (first time I've used Go) on the same system).

strace attached from running 1.60.0 (renamed to .txt to upload to github) strace.txt

[bradfitz]

Looks like it's:

4081  ioctl(15, SIOCSIFMTU, {ifr_name="tailscale0", ifr_mtu=0} <unfinished ...>
...
4081  <... ioctl resumed>)              = -1 EINVAL (Invalid argument)

And what's the strace look like when it works?

[sambartle]

Apologies I intended to attach both.. strace.txt

4595 ioctl(15, SIOCSIFMTU, {ifr_name="tailscale0", ifr_mtu=1280} <unfinished ...> 4591 nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...> 4595 <... ioctl resumed>) = 0 4595 close(15 <unfinished ...> 4591 <... nanosleep resumed>NULL) = 0 4595 <... close resumed>) = 0

So that definitly looks like a possibility.

[bradfitz]

@sambartle, can you try building this tiny Go program on your device, building it with both Go 1.21 and Go 1.22?

mtu.tar.gz

To build, from that directory:

$ GOOS=linux GOARCH=arm /path/to/go1.22/bin/go build -o mtu122
$ GOOS=linux GOARCH=arm /path/to/go1.21/bin/go build -o mtu121

Then running it on the device, pick some existing interface and run like:

nas$ sudo ./mtu122 --name=ens18 --mtu=1500
nas$ sudo ./mtu122 --name=ens18 --mtu=1499
nas$ sudo ./mtu122 --name=ens18 --mtu=1500

(and check ifconfig or ip or whatever to see that it worked?)

I'm curious if from that minimal repro we can see a difference on your device.

[bradfitz]

For the record, that code is from wireguard-go, which does:

func (tun *NativeTun) setMTU(n int) error {
    name, err := tun.Name()
    if err != nil {
        return err
    }

    // open datagram socket
    fd, err := unix.Socket(
        unix.AF_INET,
        unix.SOCK_DGRAM|unix.SOCK_CLOEXEC,
        0,
    )
    if err != nil {
        return err
    }

    defer unix.Close(fd)

    // do ioctl call
    var ifr [ifReqSize]byte
    copy(ifr[:], name)
    *(*uint32)(unsafe.Pointer(&ifr[unix.IFNAMSIZ])) = uint32(n)
    _, _, errno := unix.Syscall(
        unix.SYS_IOCTL,
        uintptr(fd),
        uintptr(unix.SIOCSIFMTU),
        uintptr(unsafe.Pointer(&ifr[0])),
    )

    if errno != 0 {
        return fmt.Errorf("failed to set MTU of TUN device: %w", errno)
    }

    return nil
}

[sambartle]

@bradfitz Apologies for the delay in taking a look at this (and thanks for the ongoing help)..

I initially tried this with the lo interface and the results were.. When built with go 1.21 your sample code behaves as expected and sets the mtu. With 1.22, the code completes successfully (errno: 0 same as with 1.21), however the mtu is set to 0 and not the requested value.

I then assumed that the interface may have something to do with it.. so i launched tailscaled (using 1.58.1), so that the tailscale0 device was created. Using that interface.. When built with go 1.21 your sample code behaves as expected and sets the mtu. With 1.22, the code errors.

sudo ./mtu122 --name=tailscale0 --mtu=1500 2024/05/03 19:21:24 errno: invalid argument

So it looks like trying to set an mtu of 0 on the TUN device is the root cause of the error (though an mtu of 0 seems like it should be invalid), for some reason code built with go 1.22 ignores the provided value and attempts to set it to 0

I also tried building with go on windows, rather than linux and the behaviour is the same with the respective go versions.