internal/wglinux: nil wg0 device returned on example cross-compiled to arm7

[schnapper79]

I use the example programm from this repository, together with boringtun. On my dekstop linux pc and on windows this program works without problems. On an arm7 device getting all created wireguard devices per c.Devices() works without problems:

interface: wg0 (userspace)
  public key: L+V9o0fNYkMVKNqsX7spBzD/9oSvxM/C7ZCZX1jLO3Q=
  private key: (hidden)
  listening port: 58097

but calling it with a named device per c.Device("wg0") i get only

interface:  (unknown)
  public key: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
  private key: (hidden)
  listening port: 0

sadly the same with configuration. I can configure my device on my desktop pc with Arch Linux (using Kernel and boringtun) and on Windows (using wireguard-go), but it doesn't work on my Arm7.

Any ideas?

[mdlayher]

It looks like you aren't checking the error return value. Can you please provide a runnable code snippet that reproduces this?

[schnapper79]

it's exactly your example in wgctrl-go/cmd/wgctrl/. I call it either by: sudo ./main or by: sudo ./main wg0

[mdlayher]

Can you please clarify the following:

• what type of device is this "arm7" machine?
• which WireGuard implementation are you using on this device: kernel, wireguard-go, boringtun, other?
• if userspace, what does ls -l /var/run/wireguard show?

There is nothing in the userspace code that would be architecture dependent AFAIK, and I'd be surprised to see the kernel interface incorrect since the genetlink headers define nothing arch-specific.

[mdlayher]

If you're building a Go program on this device, it'd be useful if you could poke around a bit and make some modifications to wgctrl-go to help me better diagnose what you're seeing.

[schnapper79]

it's an imx6 with debian stretch and a kernel 4.1.15. I am using a cross compiled "boringtun" and i do also cross compile your "main.go".

$ ls /var/run/wireguard
wg0.sock

the main surprise is: if i just call sudo ./main it shows the wg0 device with the expected configuration (unconfigured).

And yes, I can try whatever you like me to.

[mdlayher]

In that case I don't understand what could be wrong here. Can you eliminate cross compilation as a variable?

What happens if you make modifications to internal/wguser/client.go to add logging? The code paths for Devices versus Device(name) are basically identical except for some minor filtering on the input name.

[schnapper79]

Is there a way how i can check which type of interface the client is looking for? it seems that client.Device("wg0") returns an unknown type and so i don't run into internal/wguser/client.go -> Device(string)

my guess is i have to look somewhere earlier? maybe in find()?

[schnapper79]

hi.. after some digging with Printf I finally ended in internal/wglinux/client_linux.go, namly here:

// Device implements wginternal.Client.
func (c *Client) Device(name string) (*wgtypes.Device, error) {

 ... //[long function body]

    msgs, err := c.execute(wgh.CmdGetDevice, flags, b)
    if err != nil {
        fmt.Printf("Error2: %v\n" ,err)
        return nil, err
    }
    return parseDevice(msgs)
}

on my desktop the kernel wireguard interface runs through without errors, till parseDevice, gets parsed and displayed fine. The boringtun interface on this pc gets Errored out after msgs, err := c.execute(wgh.CmdGetDevice, flags, b) with Error2: file does not exist and continues in userspace implementation to display this interface correctly.

on my arm7 device, the boringtun interface is not Errored out, msgs, err := c.execute(wgh.CmdGetDevice, flags, b) returns [],nil, so msgs empty and error nil, which results in the discussed failure.

[schnapper79]

I added a check for empty []genlink.Message if error is still nil:

// execute executes a single WireGuard netlink request with the specified command,
// header flags, and attribute arguments.
func (c *Client) execute(command uint8, flags netlink.HeaderFlags, attrb []byte) ([]genetlink.Message, error) {
    msg := genetlink.Message{
        Header: genetlink.Header{
            Command: command,
            Version: wgh.GenlVersion,
        },
        Data: attrb,
    }

    msgs, err := c.c.Execute(msg, c.family.ID, flags)
    if err == nil {
        if len(msgs)==0{
                        // I expect some content in msgs or an error. If both is not true I assume os.ErrNotExist to try userspace 
            return nil, os.ErrNotExist
        }
        return msgs, nil
    }

    // We don't want to expose netlink errors directly to callers, so unpack
    // the error for use with os.IsNotExist and similar.
    oerr, ok := err.(*netlink.OpError)
    if !ok {
        // Expect all errors to conform to netlink.OpError.
        return nil, fmt.Errorf("wglinux: netlink operation returned non-netlink error (please file a bug: https://golang.zx2c4.com/wireguard/wgctrl): %v", err)
    }

    switch oerr.Err {
    // Convert "no such device" and "not a wireguard device" to an error
    // compatible with os.IsNotExist for easy checking.
    case unix.ENODEV, unix.ENOTSUP:
        return nil, os.ErrNotExist
    default:
        // Expose the inner error directly (such as EPERM).
        return nil, oerr.Err
    }
}

I have no idea if this works with other usecases, but in my case it works, so from my side we can close this issue. If you have a better idea or fix i'd be happy to hear

[mdlayher]

I have no idea how your environment is configured but I would still like to rule out cross-compilation as a variable here. Can you compile both BoringTun and wgctrl-go using that native toolchains on your ARM machine?

I can't think of any cases where it'd be valid for the netlink interface to return an empty slice of messages and nil error.

[schnapper79]

I will try to get my toolchain on my ARM device tomorrow... All I can say that now it's working perfectly so far.

[novuscy]

I think I have the same problem here.

I cross-compiled my app to run in a ARM device using a Mac.

Calling c.Device("wg0") and if wg0 does Not exist, it will return an empty Device struct and a nil error.

[jetz]

Seems same problem.

Just run cmd/wgctrl on my linux. wgctrl can list all interface correctly, but wgctrl wg0 show a confused info.

Calling c.Device("wg0") and if wg0 does Not exist, it will return an empty Device struct and a nil error.

So I can't judge whether wg0 has been existed, because c.Device("wg0") always return nil error

[jetz]

Solved by upgrading Linux kernel from 4.9.2 to 5.4.54.

[ctrlaltdel121]

I also have seen this issue on Centos 7 (Kernel 3.10.0). No cross-compiling, complied on Linux for Linux, the Execute function returns [], <nil> when looking for "wg0". Using kmod-wireguard.

Debug logs added:

calling netlink execute. msg: genetlink.Message{Header:genetlink.Header{Command:0x0, Version:0x1}, Data:[]uint8{0x8, 0x0, 0x2, 0x0, 0x77, 0x67, 0x30, 0x0}}, family 0x1b, flags: 0x301
return from netlink execute. msgs: [], err: <nil>