search

mysteriumnetwork / node

Mysterium Network Node - official implementation of distributed VPN network (dVPN) protocol
https://mysterium.network
GNU General Public License v3.0
1.11k stars 311 forks source link

Wireguard crash on macOS #2333

Closed anjmao closed 3 years ago

anjmao commented 4 years ago

I connected to WireGuard provider (hosted on Digital Ocean data center). Connection worked fine. When I pressed ctrl+C to kill myst process gracefully and mac crashed.

I was able to reproduce it most of the time following these steps:

  1. Start provider on some hosting with public IP.
  2. Connect to such provider from consumer CLI.
  3. Shutdown provider while consumer is connected.
  4. Now there should be no internet connection on consumer. Shutdown consumer node process with ctrl+c.
  5. macOS crashes.

Crash log

panic(cpu 0 caller 0xffffff80104d2f4f): assertion failed: ifp->if_sndbyte_total >= len, file: /BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.81.5/bsd/netinet/in_pcb.c, line: 3955
Backtrace (CPU 0), Frame : Return Address
0xffffff92298a39e0 : 0xffffff800fd3bb2b 
0xffffff92298a3a30 : 0xffffff800fe734d5 
0xffffff92298a3a70 : 0xffffff800fe64f4e 
0xffffff92298a3ac0 : 0xffffff800fce2a40 
0xffffff92298a3ae0 : 0xffffff800fd3b217 
0xffffff92298a3be0 : 0xffffff800fd3b5fb 
0xffffff92298a3c30 : 0xffffff80104d2aa9 
0xffffff92298a3ca0 : 0xffffff80104d2f4f 
0xffffff92298a3cb0 : 0xffffff80102f5c0e 
0xffffff92298a3cc0 : 0xffffff80102f570c 
0xffffff92298a3d10 : 0xffffff80102f5099 
0xffffff92298a3d50 : 0xffffff80102e46d9 
0xffffff92298a3d80 : 0xffffff801012395d 
0xffffff92298a3de0 : 0xffffff8010164248 
0xffffff92298a3e50 : 0xffffff8010163e68 
0xffffff92298a3e70 : 0xffffff80101226e1 
0xffffff92298a3ec0 : 0xffffff800fd7d7e5 
0xffffff92298a3f40 : 0xffffff800fd7d311 
0xffffff92298a3fa0 : 0xffffff800fce213e 

BSD process name corresponding to current thread: kernel_task

Mac OS version:
19D76

Kernel version:
Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64
Kernel UUID: A8DDE75C-CD97-3C37-B35D-1070CC50D2CE
Kernel slide:     0x000000000fa00000
Kernel text base: 0xffffff800fc00000
__HIB  text base: 0xffffff800fb00000
System model name: iMac18,3 (Mac-BE088AF8C5EB4FA2)
System shutdown begun: NO
Panic diags file available: YES (0x0)

System uptime in nanoseconds: 91984550866023
last loaded kext at 902529174527: com.parallels.kext.netbridge  15.1.4 47270 (addr 0xffffff7f94b81000, size 90112)
loaded kexts:
com.parallels.kext.netbridge    15.1.4 47270
com.parallels.kext.hypervisor   15.1.4 47270
com.parallels.kext.vnic 15.1.4 47270
org.virtualbox.kext.VBoxNetAdp  6.0.20
org.virtualbox.kext.VBoxNetFlt  6.0.20
org.virtualbox.kext.VBoxUSB 6.0.20
com.intel.kext.intelhaxm    7.5.1
org.virtualbox.kext.VBoxDrv 6.0.20
@filesystems.smbfs  3.4.1
>!ATopCaseHIDEventDriver    3430.1
>AudioAUUC  1.70
@fileutil   20.036.15
>!APlatformEnabler  2.7.0d0
>AGPM   111.4.2
>X86PlatformShim    1.0.0
@filesystems.autofs 3.0
>!AUpstreamUserClient   3.6.8
>!AHDAHardwareConfigDriver  283.15
@kext.AMDRadeonServiceManager   3.0.5
@kext.AMDFramebuffer    3.0.5
>!AHDA  283.15
@kext.AMDRadeonX4000    3.0.5
>!AGraphicsDevicePolicy 4.7.2
@AGDCPluginDisplayMetrics   4.7.2
>!AHV   1
|IOUserEthernet 1.0.1
|IO!BSerialManager  7.0.3f5
>pmtelemetry    1
>AGDCBacklightControl   4.7.2
|Broadcom!B20703USBTransport    7.0.3f5
>!A!IKBLGraphics    14.0.4
>!ABacklight    180.1
>!A!IPCHPMC 2.0.1
@Dont_Steal_Mac_OS_X    7.0.0
>eficheck   1
>!ASMCLMU   212
>!AMCCSControl  1.13
@kext.AMD9500!C 3.0.5
>ACPI_SMC_PlatformPlugin    1.0.0
>!AFIVRDriver   4.1.0
>!A!ISlowAdaptiveClocking   4.0.0
>!A!IKBLGraphicsFramebuffer 14.0.4
>!AThunderboltIP    3.1.3
>!AGFXHDA   100.1.424
|IO!BUSBDFU 7.0.3f5
>!AFileSystemDriver 3.0.1
>!AVirtIO   1.0
@filesystems.hfs.kext   522.0.9
@!AFSCompression.!AFSCompressionTypeDataless    1.0.0d1
@BootCache  40
@!AFSCompression.!AFSCompressionTypeZlib    1.0.0
@filesystems.apfs   1412.81.1
|!ABCM5701Ethernet  10.3.5
>!ASDXC 1.7.7
>AirPort.BrcmNIC    1400.1.1
@private.KextAudit  1.0
>!AAHCIPort 341.0.2
>!ARTC  2.0
>!AACPIButtons  6.1
>!ASMBIOS   2.1
>!AACPIEC   6.1
>!AAPIC 1.7
$!AImage4   1
@nke.applicationfirewall    303
$TMSafetyNet    8
@!ASystemPolicy 2.0.0
|EndpointSecurity   1
>!AHIDKeyboard  209
>!AMultitouchDriver 3430.1
>!AInputDeviceSupport   3430.1
>!AHS!BDriver   3430.1
>IO!BHIDDriver  7.0.3f5
|IOUSBUserClient    900.4.2
@kext.triggers  1.0
>DspFuncLib 283.15
@kext.OSvKernDSPLib 529
@kext.AMDRadeonX4200HWLibs  1.0
@kext.AMDRadeonX4000HWServices  3.0.5
>!AGraphicsControl  4.7.2
|IOAVB!F    800.17
>!ASSE  1.0
|Broadcom!BHost!CUSBTransport   7.0.3f5
|IO!BHost!CUSBTransport 7.0.3f5
|IO!BHost!CTransport    7.0.3f5
|IO!B!F 7.0.3f5
|IO!BPacketLogger   7.0.3f5
>!ABacklightExpert  1.1.0
>!ASMBus!C  1.0.18d1
>IOPlatformPluginLegacy 1.0.0
@!AGPUWrangler  4.7.2
|IOSlowAdaptiveClocking!F   1.0.0
>X86PlatformPlugin  1.0.0
>IOPlatformPlugin!F 6.0.0d8
>!AHDA!C    283.15
|IOHDA!F    283.15
|IOAccelerator!F2   438.3.1
>!AThunderboltEDMSink   4.2.2
>!AThunderboltDPOutAdapter  6.2.5
@kext.AMDSupport    3.0.5
@!AGraphicsDeviceControl    4.7.2
>!ASMBusPCI 1.0.14d1
|IONDRVSupport  569.4
|IOGraphics!F   569.4
@plugin.IOgPTPPlugin    810.1
>usb.IOUSBHostHIDDevice 1.2
>!UAudio    320.49
>usb.cdc    5.0.0
>usb.networking 5.0.0
>usb.!UHostCompositeDevice  1.2
|IOAudio!F  300.2
@vecLib.kext    1.2.0
|IOSerial!F 11
|IOSurface  269.6
@filesystems.hfs.encodings.kext 1
>!AThunderboltDPInAdapter   6.2.5
>!AThunderboltDPAdapter!F   6.2.5
>!AThunderboltPCIDownAdapter    2.5.4
>!AHPM  3.4.4
>!A!ILpssI2C!C  3.0.60
>!A!ILpssDmac   3.0.60
>!AXsanScheme   3
>!AThunderboltNHI   5.8.6
|IOThunderbolt!F    7.6.0
|IOEthernetAVB!C    1.1.0
|IONVMe!F   2.1.0
|IO80211!F  1200.12.2b1
>mDNSOffloadUserClient  1.0.1b8
>corecapture    1.0.4
|IOSkywalk!F    1
>!A!ILpssI2C    3.0.60
>usb.!UHostPacketFilter 1.0
|IOUSB!F    900.4.2
>!A!ILpssGspi   3.0.60
|IOAHCI!F   290.0.1
>usb.!UXHCIPCI  1.2
>usb.!UXHCI 1.2
>!AEFINVRAM 2.1
>!AEFIRuntime   2.1
|IOSMBus!F  1.1
|IOHID!F    2.0.0
$quarantine 4
$sandbox    300.0
@kext.!AMatch   1.0.0d1
>DiskImages 493.0.0
>!AFDEKeyStore  28.30
>!AEffaceable!S 1.0
>!AKeyStore 2
>!UTDM  489.80.2
|IOSCSIBlockCommandsDevice  422.0.2
>!ACredentialManager    1.0
>KernelRelayHost    1
>!ASEPManager   1.0.1
>IOSlaveProcessor   1
|IOUSBMass!SDriver  157.40.7
|IOSCSIArchitectureModel!F  422.0.2
|IO!S!F 2.1
|IOUSBHost!F    1.2
>!UHostMergeProperties  1.2
>usb.!UCommon   1.0
>!ABusPower!C   1.0
|CoreAnalytics!F    1
>!AMobileFileIntegrity  1.0.5
@kext.CoreTrust 1
|IOTimeSync!F   810.1
|IONetworking!F 3.4
|IOReport!F 47
>!AACPIPlatform 6.1
>!ASMC  3.1.9
>watchdog   1
|IOPCI!F    2.9
|IOACPI!F   1.4
@kec.pthread    1
@kec.corecrypto 1.0
@kec.Libm   1
anjmao commented 4 years ago

WireGuard native macOS app creates such network interface

utun6: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1420
options=6403<RXCSUM,TXCSUM,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
inet 10.0.44.2 --> 10.0.44.2 netmask 0xffffffff

Our node creates such interface

utun6: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1420
inet 10.182.0.2 --> 10.182.0.1 netmask 0xffffff00

It could be that we are missing whose extra options. @zolia What do you think?

I also checked ProtonVPN app and on connection it adds these options too.

ipsec0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1400
    options=6403<RXCSUM,TXCSUM,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
    inet 10.1.6.89 --> 10.1.6.89 netmask 0xffff0000
cvl commented 4 years ago

Had the same problem (same error "assertion failed: ifp->if_sndbyte_total >= len") for a while https://github.com/mysteriumnetwork/node/issues/2174

But then it disappeared. If it reappeared, perhaps would be easier to see what changed a) before May 5th, after this was fixed (May 21st).

anjmao commented 4 years ago

Tried standalone wg connection by running provider on linux and consumer on macOS. The idea was to randomly destroy wg interfaces on both ends, but wasn't able to reproduce :(

package main

import (
    "flag"
    "fmt"
    "io/ioutil"
    "log"
    "math/rand"
    "net"
    "net/http"
    "os"
    "os/exec"
    "os/signal"
    "syscall"
    "time"

    "github.com/mysteriumnetwork/node/core/port"
    "github.com/mysteriumnetwork/node/services/wireguard/endpoint"
    "github.com/mysteriumnetwork/node/services/wireguard/endpoint/kernelspace"
    "github.com/mysteriumnetwork/node/services/wireguard/endpoint/userspace"
    "github.com/mysteriumnetwork/node/services/wireguard/key"
    "github.com/mysteriumnetwork/node/services/wireguard/resources"
    "github.com/mysteriumnetwork/node/services/wireguard/wgcfg"
    "github.com/mysteriumnetwork/node/utils/netutil"
    "golang.zx2c4.com/wireguard/ipc"
)

var (
    consumerMode = flag.Bool("consumer", false, "Run as consumer")
)

const (
    providerPrivateKey = "AJ6U91UROIh/alezabeGVDO24ImWXLBJwrR33LSp+30="
    providerPublicIP   = "82.196.6.69"
    consumerPrivateKey = "aDAyenbk7Cd2/HuDGueSkCSOOWjnjQdm+ehUq1PEEWA="
)

func main() {
    flag.Parse()
    rand.Seed(time.Now().UnixNano())

    if *consumerMode {
        c := consumer{}
        go func() {
            for {
                if err := c.start(); err != nil {
                    log.Fatal(err)
                }
                ts := time.Duration(rand.Intn(10)) * time.Second
                log.Printf("Will reopen consumer in %s\n", ts)
                time.Sleep(ts)
                c.stop()
                time.Sleep(time.Second)
            }
        }()

        done := make(chan os.Signal, 1)
        signal.Notify(done, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
        <-done
        c.stop()
        return
    }

    p := provider{}
    go func() {
        for {
            if err := p.start(); err != nil {
                log.Fatal(err)
            }
            ts := time.Duration(rand.Intn(10)) * time.Second
            log.Printf("Will reopen provider in %s\n", ts)
            time.Sleep(ts)
            p.stop()
            time.Sleep(time.Second)
        }
    }()

    done := make(chan os.Signal, 1)
    signal.Notify(done, os.Interrupt, syscall.SIGINT, syscall.SIGTERM)
    <-done
    p.stop()
}

type provider struct {
    wg endpoint.WgClient

    done chan struct{}
}

func (p *provider) start() error {
    p.done = make(chan struct{}, 1)

    log.Println("Running provider mode")
    wg, err := kernelspace.NewWireguardClient()
    if err != nil {
        return fmt.Errorf("could not create wireguard client: %w", err)
    }
    p.wg = wg

    alloc := resources.NewAllocator(port.NewPool(), defaultSubnet())

    iface, err := alloc.AllocateInterface()
    if err != nil {
        return fmt.Errorf("could not allocate interface: %w", err)
    }

    peerPubKey, err := key.PrivateKeyToPublicKey(consumerPrivateKey)
    if err != nil {
        return err
    }

    subnet := defaultSubnet()
    subnet.IP = netutil.FirstIP(subnet)

    cfg := wgcfg.DeviceConfig{
        IfaceName:    iface,
        Subnet:       subnet,
        PrivateKey:   providerPrivateKey,
        ListenPort:   51820,
        DNS:          nil,
        DNSScriptDir: "",
        Peer: wgcfg.Peer{
            PublicKey:              peerPubKey,
            Endpoint:               nil,
            AllowedIPs:             []string{"0.0.0.0/0", "::/0"},
            KeepAlivePeriodSeconds: 18,
        },
    }

    errc := make(chan error, 1)
    go func() {
        log.Println("Configure device")
        errc <- wg.ConfigureDevice(cfg)
        for {
            select {
            case <-p.done:
                return
            case <-time.After(3 * time.Second):
                stats, _ := wg.PeerStats(iface)
                log.Printf("%+v", stats)
            }
        }
    }()
    if err := <-errc; err != nil {
        return fmt.Errorf("could not configure device: %w", err)
    }

    // Setup firewall.
    cmd("iptables", "-A", "POSTROUTING", "--source", "10.182.0.2/24", "!", "--destination", "10.182.0.2/24", "--jump", "SNAT", "--to", providerPublicIP, "--table", "nat")

    return nil
}

func (p *provider) stop() {
    log.Println("Stopping provider mode")
    close(p.done)
    if err := p.wg.Close(); err != nil {
        log.Fatalf("could not destroy device: %v", err)
    }
}

type consumer struct {
    wg   endpoint.WgClient
    uapi net.Listener

    done chan struct{}
}

func (c *consumer) start() error {
    c.done = make(chan struct{}, 1)

    log.Println("Running consumer mode")
    wg, err := userspace.NewWireguardClient()
    if err != nil {
        return fmt.Errorf("could not create wireguard client: %w", err)
    }
    c.wg = wg

    ports := port.NewPool()
    alloc := resources.NewAllocator(ports, defaultSubnet())

    iface, err := alloc.AllocateInterface()
    if err != nil {
        return fmt.Errorf("could not allocate interface: %w", err)
    }

    fileUAPI, err := ipc.UAPIOpen(iface)
    if err != nil {
        return fmt.Errorf("UAPI listen error: %w", err)
    }
    uapi, err := ipc.UAPIListen(iface, fileUAPI)
    if err != nil {
        return fmt.Errorf("failed to listen on UAPI socket: %w", err)
    }
    c.uapi = uapi

    go func() {
        for {
            conn, err := uapi.Accept()
            if err != nil {
                log.Println(err)
                return
            }
            go wg.Device().IpcHandle(conn)
        }
    }()

    peerPubKey, err := key.PrivateKeyToPublicKey(providerPrivateKey)
    if err != nil {
        return err
    }

    subnet := defaultSubnet()
    subnet.IP = consumerIP(subnet)

    cfg := wgcfg.DeviceConfig{
        IfaceName:    iface,
        Subnet:       subnet,
        PrivateKey:   consumerPrivateKey,
        ListenPort:   51820,
        DNS:          nil,
        DNSScriptDir: "",
        Peer: wgcfg.Peer{
            PublicKey:              peerPubKey,
            Endpoint:               &net.UDPAddr{IP: net.ParseIP(providerPublicIP), Port: 51820},
            AllowedIPs:             []string{"0.0.0.0/0", "::/0"},
            KeepAlivePeriodSeconds: 18,
        },
    }

    go c.checkIPLoop()

    errc := make(chan error, 1)
    go func() {
        log.Println("Configure device")
        errc <- wg.ConfigureDevice(cfg)

        for {
            select {
            case <-c.done:
                return
            case <-time.After(3 * time.Second):
                stats, _ := wg.PeerStats(iface)
                log.Printf("%+v", stats)
            }
        }
    }()
    if err := <-errc; err != nil {
        return fmt.Errorf("could not configure device: %w", err)
    }

    return nil
}

func (c *consumer) stop() {
    close(c.done)
    log.Println("Stopping consumer mode")
    _ = c.uapi.Close()
    if err := c.wg.Close(); err != nil {
        log.Fatalf("could not destroy device: %v", err)
    }
}

func (c *consumer) checkIPLoop() {
    for {
        select {
        case <-c.done:
            return
        case <-time.After(time.Duration(rand.Intn(7)) * time.Second):
            httpc := http.Client{Timeout: 10 * time.Second}
            resp, err := httpc.Get("https://api.ipify.org")
            if err != nil {
                log.Printf("ip fetch err: %v", err)
                time.Sleep(1 * time.Second)
                continue
            }
            defer resp.Body.Close()
            result, _ := ioutil.ReadAll(resp.Body)
            fmt.Printf("IP: %s\n", string(result))
        }
    }
}

func defaultSubnet() net.IPNet {
    return net.IPNet{
        IP:   net.ParseIP("10.182.0.0").To4(),
        Mask: net.IPv4Mask(255, 255, 0, 0),
    }
}

func consumerIP(subnet net.IPNet) net.IP {
    subnet.IP[len(subnet.IP)-1] = byte(2)
    return subnet.IP
}

func cmd(args ...string) {
    out, err := exec.Command(args[0], args[1:]...).CombinedOutput()
    if err != nil {
        log.Fatalf("could not exec command: %s:%v", string(out), err)
    }
}
stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically closed because it has not had activity for a long time. If this issue is still valid, please ping a maintainer and ask them to label it as "pinned". Thank you for your contributions.