search

costela / wesher

wireguard overlay mesh network manager
GNU General Public License v3.0
926 stars 60 forks source link

could not enable interface wgoverlay #65

Closed changchichung closed 3 years ago

changchichung commented 3 years ago

I run wesher in one of my vps , wesher can be activated normally

root@warp:/var/lib/wesher# wesher --wireguard-port 12000 --cluster-port 10000
new cluster key generated: 3W6Xn6yfuNUQX94nyPh+IC2sxCjPTeGQMH4BlLiD5AM=

but when someother nodes wants to join the cluster , then there will be an error message.

second node

chchang@hqdc039:~/git$ sudo wesher --cluster-key 3W6Xn6yfuNUQX94nyPh+abcdeCjPTeGQMH4BlLiD5AM= --join 123.123.123.123 --wireguard-port 12000 --cluster-port 10000

wesher console

root@warp:/var/lib/wesher# wesher --wireguard-port 12000 --cluster-port 10000
new cluster key generated: 3W6Xn6yfuNUQX94nyPh+IC2sxCjPTeGQMH4BlLiD5AM=
ERRO[0069] could not up interface: could not enable interface wgoverlay: address already in use 
ERRO[0110] could not up interface: could not enable interface wgoverlay: address already in use

both nodes running ubuntu focal x64 any suggestions ? or any logs I can provide ?

changchichung commented 3 years ago

more debug logs

first node

wesher log message

root@warp-abc-com:/var/lib/wesher# wesher --cluster-port 10000 --wireguard-port 12000 --log-level debug
DEBU[0000] waiting for cluster events                   
DEBU[0004] 2021/09/02 14:23:34 [DEBUG] memberlist: Stream connection from=111.222.208.84:37976 
INFO[0004] node us.abc.com joined                    
INFO[0004] cluster members:                             
INFO[0004]  addr: 111.222.208.84, overlay: {10.23.0.52 ffffffff}, pubkey: 8vczFlVOOl0wfS/wwjqeox/OjI3R1/HaZnPYH8rc8zc= 
ERRO[0004] could not up interface: could not enable interface wgoverlay: address already in use 
INFO[0004] writing entry for 10.23.0.52 ([us.abc.com]) 
DEBU[0015] 2021/09/02 14:23:45 [WARN] memberlist: Refuting a suspect message (from: us.abc.com) 
DEBU[0060] 2021/09/02 14:24:30 [WARN] memberlist: Refuting a dead message (from: us.abc.com) 
DEBU[0089] 2021/09/02 14:25:00 [DEBUG] memberlist: Initiating push/pull sync with: 111.222.208.84:10000 
DEBU[0150] 2021/09/02 14:26:00 [DEBUG] memberlist: Initiating push/pull sync with: 111.222.208.84:10000 
DEBU[0150] 2021/09/02 14:26:00 [WARN] memberlist: Refuting a suspect message (from: warp-abc-com)

second node

wesher log message

root@us:/var/lib/wesher# wesher --cluster-key 5uuWhgsth8TKL2vP7rVkcQQNjfJHYc4HAXFWhNfwMuQ= --join 111.222.145.247 --cluster-port 10000 --wireguard-port 12000 --log-level debug
DEBU[0000] 2021/09/02 14:23:34 [DEBUG] memberlist: Initiating push/pull sync with: 111.222.145.247:10000 
DEBU[0000] waiting for cluster events                   
INFO[0000] node warp-abc-com joined                  
INFO[0000] cluster members:                             
INFO[0000]  addr: 10.0.99.165, overlay: {10.121.161.244 ffffffff}, pubkey: 03j29A/uuzk/QdA3R9C1ZF/QryMfibxVRsHUSYPPUBw= 
ERRO[0000] could not up interface: could not enable interface wgoverlay: address already in use 
INFO[0000] writing entry for 10.121.161.244 ([warp-abc-com]) 
DEBU[0008] 2021/09/02 14:23:42 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0010] 2021/09/02 14:23:44 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0018] 2021/09/02 14:23:52 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0025] 2021/09/02 14:23:59 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0033] 2021/09/02 14:24:07 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0045] 2021/09/02 14:24:19 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0048] 2021/09/02 14:24:22 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0055] 2021/09/02 14:24:29 [INFO] memberlist: Marking warp-abc-com as failed, suspect timeout reached (0 peer confirmations) 
INFO[0055] node warp-abc-com left                    
INFO[0055] cluster members:                             
ERRO[0055] could not up interface: could not enable interface wgoverlay: address already in use 
INFO[0056] node warp-abc-com joined                  
INFO[0056] cluster members:                             
INFO[0056]  addr: 10.0.99.165, overlay: {10.121.161.244 ffffffff}, pubkey: 03j29A/uuzk/QdA3R9C1ZF/QryMfibxVRsHUSYPPUBw= 
ERRO[0056] could not up interface: could not enable interface wgoverlay: address already in use 
INFO[0056] writing entry for 10.121.161.244 ([warp-abc-com]) 
DEBU[0065] 2021/09/02 14:24:39 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0073] 2021/09/02 14:24:47 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0085] 2021/09/02 14:25:00 [DEBUG] memberlist: Stream connection from=111.222.145.247:47104 
DEBU[0095] 2021/09/02 14:25:09 [ERR] memberlist: Push/Pull with warp-abc-com failed: dial tcp 10.0.99.165:10000: i/o timeout 
DEBU[0095] 2021/09/02 14:25:09 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0103] 2021/09/02 14:25:17 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0125] 2021/09/02 14:25:39 [INFO] memberlist: Marking warp-abc-com as failed, suspect timeout reached (0 peer confirmations) 
INFO[0125] node warp-abc-com left                    
INFO[0125] cluster members:                             
ERRO[0125] could not up interface: could not enable interface wgoverlay: address already in use 
DEBU[0130] 2021/09/02 14:25:44 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0145] 2021/09/02 14:26:00 [DEBUG] memberlist: Stream connection from=111.222.145.247:47106 
INFO[0146] node warp-abc-com joined                  
INFO[0146] cluster members:                             
INFO[0146]  addr: 10.0.99.165, overlay: {10.121.161.244 ffffffff}, pubkey: 03j29A/uuzk/QdA3R9C1ZF/QryMfibxVRsHUSYPPUBw= 
ERRO[0146] could not up interface: could not enable interface wgoverlay: address already in use 
INFO[0146] writing entry for 10.121.161.244 ([warp-abc-com]) 
DEBU[0158] 2021/09/02 14:26:12 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0185] 2021/09/02 14:26:39 [ERR] memberlist: Push/Pull with warp-abc-com failed: dial tcp 10.0.99.165:10000: i/o timeout 
DEBU[0190] 2021/09/02 14:26:44 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0198] 2021/09/02 14:26:52 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0206] 2021/09/02 14:27:00 [DEBUG] memberlist: Stream connection from=111.222.145.247:47108
costela commented 3 years ago

could not enable interface wgoverlay: address already in use :point_up: this sounds weird. Just to be sure: have you checked that nothing else is listening on port 12000 (udp)? Maybe some previous manual tests with wireguard?

Also: does dmesg say anything useful?

changchichung commented 3 years ago

I'm pretty sure port 12000 is free. and I think this might related that I already have a interface named wg0 so I stop wg0 , then run wesher again.

first node debug messages

chchang@warp-abc-com:~$ sudo wesher --wireguard-port 12000 --cluster-port 10000 --log-level debug
DEBU[0000] 2021/09/03 20:39:40 [DEBUG] memberlist: Initiating push/pull sync with: 111.111.111.111:10000 
DEBU[0000] 2021/09/03 20:39:40 [WARN] memberlist: Refuting a suspect message (from: warp-abc-com) 
DEBU[0000] waiting for cluster events                   
INFO[0000] node ws.abc.com joined                    
INFO[0000] cluster members:                             
INFO[0000]  addr: 111.111.111.111, overlay: {10.15.5.130 ffffffff}, pubkey: FOT8uDJsptwZvtSR+whGyG/fZ6m6/S/JDdWbDnO9MnU= 
INFO[0000] writing entry for 10.15.5.130 ([ws.abc.com]) 
DEBU[0015] 2021/09/03 20:39:55 [WARN] memberlist: Refuting a suspect message (from: ws.abc.com) 
DEBU[0035] 2021/09/03 20:40:15 [WARN] memberlist: Refuting a suspect message (from: ws.abc.com) 
DEBU[0060] 2021/09/03 20:40:40 [WARN] memberlist: Refuting a suspect message (from: ws.abc.com) 
DEBU[0119] 2021/09/03 20:41:39 [DEBUG] memberlist: Initiating push/pull sync with: 111.111.111.111:10000 
DEBU[0120] 2021/09/03 20:41:40 [WARN] memberlist: Refuting a suspect message (from: warp-abc-com) 
DEBU[0160] 2021/09/03 20:42:20 [WARN] memberlist: Refuting a suspect message (from: ws.abc.com) 
DEBU[0180] 2021/09/03 20:42:40 [DEBUG] memberlist: Initiating push/pull sync with: 111.111.111.111:10000

second node debug messages

sudo ./wesher --cluster-key /7B6D2zT1VIg5KTF975lHS4H5n6fwvfmYr/gwM/ht7M= --join 222.222.222.222 --cluster-port 10000 --wireguard-port 12000 --log-level debug
DEBU[0000] 2021/09/03 20:39:13 [DEBUG] memberlist: Initiating push/pull sync with: 222.222.222.222:10000 
DEBU[0000] waiting for cluster events                   
INFO[0000] node warp-abc-com joined                  
INFO[0000] cluster members:                             
INFO[0000]  addr: 10.0.99.165, overlay: {10.121.161.244 ffffffff}, pubkey: tM3W37I6NVP8lwIh3U5NlYLsQQu7QbCCdyUn1CTaCRY= 
INFO[0000] writing entry for 10.121.161.244 ([warp-abc-com]) 
DEBU[0008] 2021/09/03 20:39:21 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0010] 2021/09/03 20:39:23 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
INFO[0014] node warp-abc-com left                    
INFO[0014] cluster members:                             
DEBU[0026] 2021/09/03 20:39:40 [DEBUG] memberlist: Stream connection from=222.222.222.222:52712 
INFO[0027] node warp-abc-com joined                  
INFO[0027] cluster members:                             
INFO[0027]  addr: 10.0.99.165, overlay: {10.121.161.244 ffffffff}, pubkey: /br5D8MvVbfsj5AJ7cni7jJXGSv6DEmsfajcsu1zLXc= 
INFO[0027] writing entry for 10.121.161.244 ([warp-abc-com]) 
DEBU[0033] 2021/09/03 20:39:46 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0040] 2021/09/03 20:39:53 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received 
DEBU[0048] 2021/09/03 20:40:01 [DEBUG] memberlist: Failed ping: warp-abc-com (timeout reached) 
DEBU[0060] 2021/09/03 20:40:13 [INFO] memberlist: Suspect warp-abc-com has failed, no acks received

althrough there are still some timeout/failed message I still notice that /etc/hosts has changed and I can ping to each other

first node /etc/hosts and ping second node result
2021-09-03 20:42:29 [chchang@ws ~]$ cat /etc/hosts

127.0.0.1    localhost
127.0.1.1    ws.abc.com

# The following lines are desirable for IPv6 capable hosts
::1          localhost ip6-localhost ip6-loopback
ff02::1      ip6-allnodes
ff02::2      ip6-allrouters

10.121.161.244  warp-abc-com    # ! MANAGED AUTOMATICALLY !
2021-09-03 20:43:56 [chchang@ws ~]$
second node /etc/hosts and ping first node result
chchang@warp-abc-com:~$ cat /etc/hosts
127.0.0.1   localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost   ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
127.0.1.1   warp-abc-com    warp-abc-com

10.15.5.130 ws.abc.com  # ! MANAGED AUTOMATICALLY !
chchang@warp-abc-com:~$ ping ws.abc.com
PING ws.abc.com (10.15.5.130) 56(84) bytes of data.
64 bytes from ws.abc.com (10.15.5.130): icmp_seq=1 ttl=64 time=142 ms
64 bytes from ws.abc.com (10.15.5.130): icmp_seq=2 ttl=64 time=142 ms
64 bytes from ws.abc.com (10.15.5.130): icmp_seq=3 ttl=64 time=142 ms
64 bytes from ws.abc.com (10.15.5.130): icmp_seq=4 ttl=64 time=142 ms
^C
--- ws.abc.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 142.009/142.138/142.267/0.114 ms
chchang@warp-abc-com:~$

I think wesher works as expect now . and I will try to make this peer-to-peer VPN to site-to-site VPN

costela commented 3 years ago

I think wesher works as expect now .

That's weird, because it should not have any issues with an existing wg0 interface. Maybe some firewall filtering UDP?

I will try to make this peer-to-peer VPN to site-to-site VPN

That's probably not going to work out-of-the-box because of #28 :unamused: