search

zerotier / ZeroTierOne

A Smart Ethernet Switch for Earth
https://zerotier.com
Other
14.25k stars 1.66k forks source link

A Strange Connection Issue #1411

Closed ghost closed 3 years ago

ghost commented 3 years ago

ABOUT THE ISSUE

ENVIRONMENT

I have three PCs with installed ZT1 software. PC Short Name WAN Network LAN Network OS ZT1 Installed
Huawei Server HWS Huawei IDC BGP LAN-1 Debian 10 1.6.5
Linux Server LIN China Mobile LAN-2 Debian 10 1.6.5 in Docker
Windows PC WIN China Mobile LAN-2 Windows 10 1809 1.4.6
PC Short Name NAT Type Public IPv4 Public IPv6 Ports Open on Router
Huawei Server HWS Symmetric Yes No TCP/UDP 9993 for IPv4
Linux Server LIN Fullcone No Yes All for IPv6
TCP/UDP 9993 for IPv4
UDP 41097 for IPv4
Windows PC WIN Fullcone No Yes All for IPv6
UDP 53270 for IPv4

TOPOLOGICAL

WHATS THE PROBLEM

Before Someday ❤️

Connection (A <--> B) Latency Speed Peer A to Peer B Peer B to Peer A
LIN <--> HWS < 100ms Good DIRECT DIRECT
WIN <--> HWS < 100ms Good DIRECT DIRECT

After Someday 🙉

All the environment keeps no change, did only apt-get upgrade on the HWS and LIN.
But the stranger connection issue was happend.		 Connection (A <--> B) Latency Speed Peer A to Peer B Peer B to Peer A
LIN <--> HWS > 300ms Bad DIRECT RELAY
WIN <--> HWS < 100ms Good DIRECT DIRECT

AHAaaa ...

I dont think the issue is related to the network environment🤔 due to eveything works fine before someday🤕 what cause it.. ❓😄

erikh commented 3 years ago

Do you know what day the issue started?

On Thu, Jun 10, 2021 at 12:58 AM Pekaikon Norckon @.***> wrote:

ABOUT THE ISSUE ENVIRONMENT

I have three PCs with installed ZT1 software. PC Short Name WAN Network LAN Network OS ZT1 Installed Huawei Server HWS Huawei IDC BGP LAN-1 Debian 10 1.6.5 Linux Server LIN China Mobile LAN-2 Debian 10 1.6.5 in Docker Windows PC WIN China Mobile LAN-2 Windows 10 1809 1.4.6 PC Short Name NAT Type Public IPv4 Public IPv6 Ports Open Huawei Server HWS Symmetric Yes No TCP/UDP 9993 for IPv4 Linux Server LIN Fullcone No Yes All for IPv6 TCP/UDP 9993 for IPv4 UDP 41097 for IPv4 Windows PC WIN Fullcone No Yes All for IPv6 UDP 53270 for IPv4 TOPOLOGICAL

https://camo.githubusercontent.com/be2650722b6f0ef81af5ee4816c16a9fb6a4007ca4adbc0e6fbe4f350dd48efa/68747470733a2f2f692e6c6f6c692e6e65742f323032312f30362f31302f524463626654416a576b694c4265762e706e67 WHATS THE PROBLEM Before Someday ❤️ Connection (A <--> B) Latency Speed Peer A to Peer B Peer B to Peer A LIN <--> HWS < 100ms Good DIRECT DIRECT WIN <--> HWS < 100ms Good DIRECT DIRECT After Someday 🙉

All the environment keeps no change, did only apt-get upgrade on the HWS and LIN. But the stranger connection issue was happend. Connection (A <--> B) Latency Speed Peer A to Peer B Peer B to Peer A LIN <--> HWS > 300ms Bad DIRECT RELAY WIN <--> HWS < 100ms Good DIRECT DIRECT AHAaaa ...

I dont think the issue is related to the network environment🤔 due to eveything works fine before someday🤕 what cause it.. ❓😄

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/zerotier/ZeroTierOne/issues/1411, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAET27JCUGWUKP5TLS76YDTSBWCFANCNFSM46NR2MEQ .

ghost commented 3 years ago

@erikh sorry i forget when it started. 😹

but in yesterday, i moved PC WIN to third ISP (China Unicom), WIN<->LIN has BAD network in RELAY mode, WIN<->HWS works great in DIRECT mode.

maybe the issue related in Debian Linux, ZT1 or Docker but not related with WAN Network?

laduke commented 3 years ago

Try the work around described here https://discuss.zerotier.com/t/coma-bug-update/3138

Closing because there isn't a bug we can reproduce and fix, but we can keep talking here.

ghost commented 3 years ago

@laduke thank you for reply, i tried remove identify.public and identify.secret then restart zerotier-one. the port changed from 41097 to 48823. but the issue still exist.

on Linux PC in LAN-2

All the PLANET is in DIRECT mode, to HWS in RELAY mode

<ztaddr>   <ver>  <role> <lat> <link> <lastTX> <lastRX> <path>
26d962bfe5 1.4.6  LEAF      27 DIRECT 4880     4880     ***
35108ab5f8 1.4.6  LEAF      51 DIRECT 6090     6090     ***
6027a852f1 1.6.5  LEAF      40 DIRECT 438      438      ***
61d294b9cb -      PLANET   206 DIRECT 2547     2341     2001:49f0:d002:6::2/9993
62f865ae71 -      PLANET    76 DIRECT 7587     7511     50.7.252.138/9993
6bf1c94aea 1.4.6  LEAF     148 DIRECT 6830     6830     ***
778cde7190 -      PLANET   250 DIRECT 2548     2298     103.195.103.66/9993
92dc882a64 1.4.6  LEAF      18 DIRECT 15236    16527    ***
93afae5963 1.6.4  LEAF     233 DIRECT 8802     8802     ***
988f1f1da6 1.6.2  LEAF     141 DIRECT 2414     2414     ***
992fcf1db7 -      PLANET   262 DIRECT 2548     2290     195.181.173.159/9993
a09acf0233 1.6.4  LEAF     244 DIRECT 2548     2307     ***
c2fef121ce 1.6.5  LEAF      96 DIRECT 9802     9802     ***
cd2e836f10 1.6.2  LEAF      57 DIRECT 6255     6255     ***
f2f1***edd 1.6.4  LEAF      -1 RELAY    <----- this is HWS

on Windows PC in LAN-2 (same router with Linux PC)

to HWS is in DIRECT mode.

<ztaddr>   <ver>  <role> <lat> <link> <lastTX> <lastRX> <path>
61d294b9cb -      PLANET   415 DIRECT 14396    3412     2001:49f0:d002:6::2/9993
62f865ae71 -      PLANET   145 DIRECT 3176     2240     50.7.252.138/9993
6bf1c94aea 1.4.6  LEAF      76 DIRECT 36330    39332    ***
778cde7190 -      PLANET   412 DIRECT 14398    3292     2605:9880:400:c3:254:f2bc:a1f7:19/9993
93afae5963 1.6.4  LEAF    1654 DIRECT 3322     2826     ***
988f1f1da6 1.6.2  LEAF      -1 DIRECT 54       54       ***
992fcf1db7 -      PLANET   508 DIRECT 3828     3316     195.181.173.159/9993
a09acf0233 1.6.4  LEAF    -468 DIRECT 13614    13614   ***
c2fef121ce 1.6.5  LEAF      74 DIRECT 19732    21554    ***
cd2e836f10 1.6.2  LEAF      70 DIRECT 19732    2442     ***
dc5506a6f9 1.6.5  LEAF       4 DIRECT 14400    14400    ***
f2f1***edd 1.6.5  LEAF     102 DIRECT 1264     1264     <--- this is HWS

my router is OpenWRT with kernel 5.4.113. my linux pc is Debian 10 with kernel 4.19.0-16-amd64

ghost commented 3 years ago

@laduke @erikh this problem was solved due to docker contianer. moving zerotier from docker to host and the issue can be solve😄 thanks everyone😸

Docker Container Inspect Dump

{
    "AppArmorProfile": "docker-default",
    "Args": [],
    "Config": {
        "AttachStderr": false,
        "AttachStdin": false,
        "AttachStdout": false,
        "Cmd": [
            "zerotier-one"
        ],
        "Domainname": "",
        "Entrypoint": null,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "ZEROTIER_VERSION=1.6.2",
            "ZEROTIER_COMMIT=e4404164bd9eb14c91906ec3cf577ba98eb24b8a"
        ],
        "ExposedPorts": {
            "9993/udp": {}
        },
        "Hostname": "isu-73100-1",
        "Image": "lifeym/zerotier:latest",
        "Labels": {
            "description": "Containerized ZeroTier One for use on Linux hosts.",
            "maintainer": "Containerized ZeroTier One Maintainers <leonardo_yu@hotmail.com>",
            "org.label-schema.build-date": "2020-12-05T00:00:00Z",
            "org.label-schema.description": "Containerized ZeroTier One for use on Linux hosts.",
            "org.label-schema.docker.cmd": "docker run --device=/dev/net/tun     --net=host     --cap-add=NET_ADMIN     --cap-add=SYS_ADMIN     --cap-add=CAP_SYS_RAWIO     -v /path/to/my-zerotier-one:/var/lib/zerotier-one     --name zerotier-one     -d lifeym/zerotier",
            "org.label-schema.name": "zerotier",
            "org.label-schema.schema-version": "1.0",
            "org.label-schema.url": "https://zerotier.com",
            "org.label-schema.version": "{1.6.2}",
            "version": "1.6.2"
        },
        "OnBuild": null,
        "OpenStdin": false,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": {
            "/var/lib/zerotier-one": {}
        },
        "WorkingDir": ""
    },
    "Created": "2021-06-10T09:17:44.370651288Z",
    "Driver": "overlay2",
    "ExecIDs": null,
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/22bb38b61b8a6c9d356188fe3f41ea9ef87b204604e1996ddcfd615bdde4b8e5-init/diff:/var/lib/docker/overlay2/ee173896e37eba6a9c71102e43088b53d03a5215043d957901524b3bb5627e7b/diff:/var/lib/docker/overlay2/cb497a402268d0f92d8f70f22bd6942d65e3e4f000de969e55ea4f269c068726/diff:/var/lib/docker/overlay2/ed3a1aa029cdd25e151e6dec84d23a16bb8dd673f7332907a231b2d8d096ce53/diff",
            "MergedDir": "/var/lib/docker/overlay2/22bb38b61b8a6c9d356188fe3f41ea9ef87b204604e1996ddcfd615bdde4b8e5/merged",
            "UpperDir": "/var/lib/docker/overlay2/22bb38b61b8a6c9d356188fe3f41ea9ef87b204604e1996ddcfd615bdde4b8e5/diff",
            "WorkDir": "/var/lib/docker/overlay2/22bb38b61b8a6c9d356188fe3f41ea9ef87b204604e1996ddcfd615bdde4b8e5/work"
        },
        "Name": "overlay2"
    },
    "HostConfig": {
        "AutoRemove": false,
        "Binds": [
            "/var/lib/zerotier-one:/var/lib/zerotier-one"
        ],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceWriteIOps": null,
        "BlkioWeight": 0,
        "BlkioWeightDevice": [],
        "CapAdd": [
            "AUDIT_WRITE",
            "CHOWN",
            "DAC_OVERRIDE",
            "FOWNER",
            "FSETID",
            "KILL",
            "MKNOD",
            "NET_ADMIN",
            "NET_BIND_SERVICE",
            "NET_RAW",
            "SETFCAP",
            "SETGID",
            "SETPCAP",
            "SETUID",
            "SYS_ADMIN",
            "SYS_CHROOT"
        ],
        "CapDrop": [
            "AUDIT_CONTROL",
            "BLOCK_SUSPEND",
            "DAC_READ_SEARCH",
            "IPC_LOCK",
            "IPC_OWNER",
            "LEASE",
            "LINUX_IMMUTABLE",
            "MAC_ADMIN",
            "MAC_OVERRIDE",
            "NET_BROADCAST",
            "SYSLOG",
            "SYS_BOOT",
            "SYS_MODULE",
            "SYS_NICE",
            "SYS_PACCT",
            "SYS_PTRACE",
            "SYS_RAWIO",
            "SYS_RESOURCE",
            "SYS_TIME",
            "SYS_TTY_CONFIG",
            "WAKE_ALARM"
        ],
        "Cgroup": "",
        "CgroupParent": "",
        "CgroupnsMode": "host",
        "ConsoleSize": [
            0,
            0
        ],
        "ContainerIDFile": "",
        "CpuCount": 0,
        "CpuPercent": 0,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpuShares": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "Devices": [
            {
                "CgroupPermissions": "rwm",
                "PathInContainer": "/dev/net/tun",
                "PathOnHost": "/dev/net/tun"
            }
        ],
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": [],
        "GroupAdd": null,
        "IOMaximumBandwidth": 0,
        "IOMaximumIOps": 0,
        "IpcMode": "private",
        "Isolation": "",
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "Links": null,
        "LogConfig": {
            "Config": {},
            "Type": "json-file"
        },
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "Memory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "NanoCpus": 0,
        "NetworkMode": "host",
        "OomKillDisable": false,
        "OomScoreAdj": 0,
        "PidMode": "",
        "PidsLimit": null,
        "PortBindings": {},
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ],
        "ReadonlyRootfs": false,
        "RestartPolicy": {
            "MaximumRetryCount": 0,
            "Name": "unless-stopped"
        },
        "Runtime": "runc",
        "SecurityOpt": null,
        "ShmSize": 67108864,
        "UTSMode": "",
        "Ulimits": null,
        "UsernsMode": "",
        "VolumeDriver": "",
        "VolumesFrom": null
    },
    "HostnamePath": "/var/lib/docker/containers/1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95/hostname",
    "HostsPath": "/var/lib/docker/containers/1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95/hosts",
    "Id": "1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95",
    "Image": "sha256:7a0a27940f5db178f0826aa6758fc0032c21319821b52e1ae43aefcd2a96ead9",
    "LogPath": "/var/lib/docker/containers/1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95/1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95-json.log",
    "MountLabel": "",
    "Mounts": [
        {
            "Destination": "/var/lib/zerotier-one",
            "Mode": "",
            "Propagation": "rprivate",
            "RW": true,
            "Source": "/var/lib/zerotier-one",
            "Type": "bind"
        }
    ],
    "Name": "/zerotier-one",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Networks": {
            "host": {
                "Aliases": null,
                "DriverOpts": null,
                "EndpointID": "",
                "Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": {},
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "",
                "NetworkID": "266c2ac8926ded592758672227a48da60c0785e79802ddbdef2c0d076abbef1a"
            }
        },
        "Ports": {},
        "SandboxID": "723f893cfa607df4f7d5b27f08790f5078c1c06fd555fe791d223f8a78b1ba6e",
        "SandboxKey": "/var/run/docker/netns/default",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },
    "Path": "zerotier-one",
    "Platform": "linux",
    "Portainer": {
        "ResourceControl": {
            "Id": 63,
            "ResourceId": "1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95",
            "SubResourceIds": [],
            "Type": 1,
            "UserAccesses": [],
            "TeamAccesses": [],
            "Public": false,
            "AdministratorsOnly": true,
            "System": false
        }
    },
    "ProcessLabel": "",
    "ResolvConfPath": "/var/lib/docker/containers/1b94d68265e807e30dbd449c8869d00b6c19c7c5cfd7085f974725ac8f2cdb95/resolv.conf",
    "RestartCount": 0,
    "State": {
        "Dead": false,
        "Error": "",
        "ExitCode": 137,
        "FinishedAt": "2021-06-13T03:53:21.304302426Z",
        "OOMKilled": false,
        "Paused": false,
        "Pid": 0,
        "Restarting": false,
        "Running": false,
        "StartedAt": "2021-06-13T02:32:10.80765532Z",
        "Status": "exited"
    }
}