zerotier / ZeroTierOne

A Smart Ethernet Switch for Earth
https://zerotier.com
Other
14.31k stars 1.67k forks source link

zerotierone doesn't create dev zt0 on debian 8 with OpenVPN-Server installed. #699

Closed quetsch closed 6 years ago

quetsch commented 6 years ago

Hi! I open up a new issue because 2 other threads with similiar issues were closed without a solution: https://github.com/zerotier/ZeroTierOne/issues/497 https://github.com/zerotier/ZeroTierOne/issues/448 There is definitely an issue with creation of a zt0 interface on Debian 8 with openvpn server installed. The error message in /var/log/syslog is as follows: zerotier-one[378]: ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory.

I installed zerotier-one freshly on two machines, a local LAN server and on a virtual server, both running debian 8 (uname -r): LAN-Server: 3.16.0-5-amd64 vServer: 3.16.0 The vServer is configured as openvpn server with both a tun and a tap interface. However, joining my private network works on both machines, no traffic to the vServer however (PORT_ERROR) sudo zerotier-cli listnetworks 200 listnetworks 200 listnetworks a09acf02333f90c3 Quetsch c2:26:be:0f:c7:29 PORT_ERROR PRIVATE fc93:a55f:c1b6:813c:c5e6:0000:0000:0001/40,10.100.79.1/24

Any help would be appreciated. BTW: No change when I shut down openvpn and the tun/tap interfaces go down before installation. Seems like an issue in coexisting with openvpn.

PS: a similar issue was reported on centos7 here, thread closed. If I can provide any more information, I am glad to help.

janjaapbos commented 6 years ago

Does /dev/net/tun exist? Is it perhaps moved somewhere else in combination with OpenVPN?

quetsch commented 6 years ago

Of course it exists on both machines: sudo ls /dev/net/ tun

sudo ifconfig `lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:25 errors:0 dropped:0 overruns:0 frame:0 TX packets:25 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes:2224 (2.1 KiB) TX bytes:2224 (2.1 KiB)

tap0 Link encap:Ethernet HWaddr ee:05:76:13:a5:8b inet addr:10.19.80.1 Bcast:10.19.80.255 Mask:255.255.255.0 inet6 addr: fe80::ec05:76ff:fe13:a58b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:38 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:6728 (6.5 KiB)

tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00 inet addr:10.19.81.1 P-t-P:10.19.81.2 Mask:255.255.255.255 inet6 addr: fe80::9e1f:ed86:c3a2:3c28/64 Scope:Link UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1400 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 B) TX bytes:144 (144.0 B)

venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00 inet addr:127.0.0.1 P-t-P:127.0.0.1 Bcast:0.0.0.0 Mask:255.255.255. 255 inet6 addr: ::2/128 Scope:Compat UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1 RX packets:172848 errors:0 dropped:0 overruns:0 frame:0 TX packets:171118 errors:0 dropped:7207 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:17150843 (16.3 MiB) TX bytes:18117412 (17.2 MiB)

venet0:0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 -00 inet addr:178.X.X.X P-t-P:178.X.X.X Bcast:178.X.X.255 Ma sk:255.255.255.0 UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1 `

As stated: OpenVPN works fine with 2 profiles, 1 for tun, 1 for tap. The interface zt0 just doesn't get created by the installer. It must habe something to do with openvpn because a nearly identical other machine with no openvpn installed does not have that issue.

quetsch commented 6 years ago

And as mentioned in the other threads, apparmor or selinux is not installed/used as far as I can see: sudo service apparmor status ● apparmor.service Loaded: not-found (Reason: No such file or directory) Active: inactive (dead) sudo check-selinux-installation sudo: check-selinux-installation: command not found sudo selinux-activate sudo: selinux-activate: command not found

adamierymenko commented 6 years ago

This really kind of makes no sense. The kernel tap device driver supports thousands of devices and there isn't any restriction about which processes can open them.

Can you try shutting down the service and then running /usr/sbin/zerotier-one manually (via sudo) and telling us if it prints anything?

quetsch commented 6 years ago

Hi! Sorry, was on easter vacation. After stopping the service, the same error appears: First stopping service:

sudo /etc/init.d/zerotier-one stop [ ok ] Stopping zerotier-one (via systemctl): zerotier-one.service. sudo /etc/init.d/zerotier-one status ● zerotier-one.service - ZeroTier One Loaded: loaded (/lib/systemd/system/zerotier-one.service; enabled) Active: inactive (dead) since Mon 2018-04-02 03:41:54 UTC; 1min 48s ago Process: 14383 ExecStart=/usr/sbin/zerotier-one (code=exited, status=0/SUCCESS) Main PID: 14383 (code=exited, status=0/SUCCESS)

Apr 02 03:41:52 vXXXXX.1blu.de systemd[1]: Started ZeroTier One. Apr 02 03:41:52 vXXXXX.1blu.de zerotier-one[14383]: ERROR: unable to configur... Apr 02 03:41:54 vXXXXX.1blu.de systemd[1]: Stopping ZeroTier One... Apr 02 03:41:54 vXXXXX.1blu.de systemd[1]: Stopped ZeroTier One. Apr 02 03:43:39 vXXXXX.1blu.de systemd[1]: Stopped ZeroTier One. Hint: Some lines were ellipsized, use -l to show in full.

(I "Xed" the exact hostname, this forum is public.)

Then the command: sudo /usr/sbin/zerotier-one ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory

Can I provide anything else to help? It is definitely an issue with an already installed OpenVPN.

adamierymenko commented 6 years ago

Is SELinux enabled? Maybe there's a rule or permission problem.

quetsch commented 6 years ago

Hi! Sorry for the late reply. AND: As I already mentioned, no SELinux is NOT enabled, as well no apparmor is in use: sudo sestatus sudo: sestatus: command not found selinuxenabled -bash: selinuxenabled: command not found sudo selinuxenabled command not found sudo cat /etc/sysconfig/selinux cat: /etc/sysconfig/selinux: No such file or directory

adamierymenko commented 6 years ago

I really don't know then... we use it alongside other things and I have never seen this issue. Linux has no limit on the number of tun/tap devices.

Can you shut down the ZeroTier service and try running it manually with "sudo /usr/sbin/zerotier-one"? See what it outputs and if there are any meaningful error messages.

quetsch commented 6 years ago

sudo service zerotier-one stop sudo service zerotier-one status ● zerotier-one.service - ZeroTier One Loaded: loaded (/lib/systemd/system/zerotier-one.service; enabled) Active: inactive (dead) since Wed 2018-04-25 12:42:41 UTC; 4s ago Process: 378 ExecStart=/usr/sbin/zerotier-one (code=exited, status=0/SUCCESS) Main PID: 378 (code=exited, status=0/SUCCESS)

Apr 23 23:45:24 v65274.1blu.de systemd[1]: Started ZeroTier One. Apr 23 23:45:24 v65274.1blu.de zerotier-one[378]: ERROR: unable to configure ... Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopping ZeroTier One... Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopped ZeroTier One. Hint: Some lines were ellipsized, use -l to show in full.

sudo /usr/sbin/zerotier-one ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory (Same as written 3 posts above).

I know it's a strange error. I can manually create tun/tap interfaces with the help of ip: sudo ip tuntap add name tap0 mode tap sudo ip link show

Now the thread is marked as "cantreproduce". I wonder if you installed OpenVPN prior to zerotier-one and also configured to use a TAP and a TUN device (see my 2nd post)?

Now I am just guessing. Can the error be related to venet0-00 network devices instead of eth0-devices?

However it's a bit frustrating answer the same questions over and over again with the same result. I know it ist an Open Source project and the support here is voluntarily, but I slowly get the impression that after asking the top 5 standard issues you are out of ideas and the threads' gonna die somehow.

So, can it have something to do with venet-0 devices on a virtual server, maybe in the routine on how tun/tap devices are created? It is possible via the "ip" command or with "openvpn -mktun".

Is there a way to increase verbosity level for logs???

quetsch commented 6 years ago

sudo service zerotier-one stop sudo service zerotier-one status ● zerotier-one.service - ZeroTier One Loaded: loaded (/lib/systemd/system/zerotier-one.service; enabled) Active: inactive (dead) since Wed 2018-04-25 12:42:41 UTC; 4s ago Process: 378 ExecStart=/usr/sbin/zerotier-one (code=exited, status=0/SUCCESS) Main PID: 378 (code=exited, status=0/SUCCESS)

Apr 23 23:45:24 v65274.1blu.de systemd[1]: Started ZeroTier One. Apr 23 23:45:24 v65274.1blu.de zerotier-one[378]: ERROR: unable to configure ... Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopping ZeroTier One... Apr 25 12:42:41 v65274.1blu.de systemd[1]: Stopped ZeroTier One. Hint: Some lines were ellipsized, use -l to show in full.

sudo /usr/sbin/zerotier-one ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory (Same as written 3 posts above).

I know it's a strange error. I can manually create tun/tap interfaces with the help of ip: sudo ip tuntap add name tap0 mode tap sudo ip link show

Now the thread is marked as "cantreproduce". I wonder if you installed OpenVPN prior to zerotier-one and also configured to use a TAP and a TUN device (see my 2nd post)?

Now I am just guessing. Can the error be related to venet0-00 network devices instead of eth0-devices?

However it's a bit frustrating answer the same questions over and over again with the same result. I know it ist an Open Source project and the support here is voluntarily, but I slowly get the impression that after asking the top 5 standard issues you are out of ideas and the threads' gonna die somehow.

So, can it have something to do with venet-0 devices on a virtual server, maybe in the routine on how tun/tap devices are created? It is possible via the "ip" command or with "openvpn -mktun".

Is there a way to increase verbosity level for logs???

quetsch commented 6 years ago

Issue still persisting in 1.2.8

maxnowack commented 6 years ago

Same issue here …

s-frostick commented 6 years ago

So i'm not sure if this will help but i was experiencing the same problem, i did an strace of the zerotier process.

close(9)                                = 0
brk(0xc24000)                           = 0xc24000
open("/dev/net/tun", O_RDWR)            = -1 EACCES (Permission denied) 
open("/dev/tun", O_RDWR)                = -1 ENOENT (No such file or directory)
brk(0xc2d000)                           = 0xc2d000
brk(0xc36000)                           = 0xc36000 
writev(2, [{"ERROR: unable to configure virtu"..., 49}, {"could not open TUN/TAP device: N"..., 56}],   2ERROR: unable to configure virtual network port: could not open TUN/TAP device: No such file or directory) = 105

So i checked the permission of /dev/net/tun

ls -la /dev/net/tun 
crw-rw---- 1 root 413 10, 200 Jun  5 01:08 /dev/net/tun

Now setting the permission to 0666 fixed the "No such file or directory" error for me.

https://www.kernel.org/doc/Documentation/networking/tuntap.txt

Set permissions: e.g. chmod 0666 /dev/net/tun There's no harm in allowing the device to be accessible by non-root users, since CAP_NET_ADMIN is required for creating network devices or for connecting to network devices which aren't owned by the user in question. If you want to create persistent devices and give ownership of them to unprivileged users, then you need the /dev/net/tun device to be usable by those users.

maxnowack commented 6 years ago

Thanks @s-frostick! Setting the permissions to 0666 fixed the issue for me as well 😊

laduke commented 6 years ago

why is the user "1" ?

s-frostick commented 6 years ago

@laduke the user is root the number you are referencing is the number of hard links to the file.

https://www.debian.org/doc/manuals/debian-reference/ch01.en.html#_links

laduke commented 6 years ago

Oops, off by one. (group is 413)

quetsch commented 6 years ago

Hi! I checked the above on my both machines, one where zerotier is working, one where it is not (both Debian Jessie). The permissions seem identical:

ZT working: ls -la /dev/net/tun crw-rw-rw- 1 root root 10, 200 Jun 7 13:56 /dev/net/tun sudo ls -la /dev/net/tun crw-rw-rw- 1 root root 10, 200 Jun 7 13:56 /dev/net/tun

ZT not working: ls -la /dev/net/tun ls: cannot access /dev/net/tun: Permission denied sudo ls -la /dev/net/tun crw-rw-rw- 1 root root 10, 200 May 31 00:16 /dev/net/tun

The file permissions are obivously the same. However, I noticed on the machine where the issue persists, I can't "ls -la /dev/net/tun" as a normal user, on the other machine I can. Well, I think that has nothing to do with my issue.

Still no zt0 interface is created: sudo zerotier-cli listnetworks 200 listnetworks 200 listnetworks a09acf02333f90c3 Quetsch c2:26:be:0f:c7:29 PORT_ERROR PRIVATE fc93:a55f:c1b6:813c:c5e6:0000:0000:0001/40,10.100.79.1/24

strace behaves similar at the system with the issue: 12433 close(9) = 0 12433 chmod("/var/lib/zerotier-one/networks.d/a09acf02333f90c3.conf", 0600) = 0 12433 brk(0xed8000) = 0xed8000 12433 open("/dev/net/tun", O_RDWR) = -1 EACCES (Permission denied) 12433 open("/dev/tun", O_RDWR) = -1 ENOENT (No such file or directory) 12433 brk(0xee1000) = 0xee1000 12433 brk(0xeea000) = 0xeea000 12433 writev(2, [{"ERROR: unable to configure virtu"..., 49}, {"could not open TUN/TAP device: N"..., 56}], 2) = 105

Well, it seems like a permission issue, but chmod 666 or even chmod 777 on /dev/net/tun doesn't change it...

factormystic commented 6 years ago

FYI I found this issue via google after following the directions for getting started with docker in the knowledgebase article here. chmod 0666 /dev/net/tun did work for me.

joseph-henry commented 6 years ago

Is anyone still experiencing this issues as of 1.2.12? It looks like a working solution has been found for at least a couple of those reporting the issue. I'm going to close this ticket for now but feel free to request that we re-open it.

NeedsCoffee commented 5 years ago

I just encountered this on v1.2.12 The chmod fix helped me and I had installed into a Scaleway VM that was running Debian 9 Permissions on /dev/net/tun were previously: crw------- Afterwards permissions were: crw-rw-rw-

quetsch commented 5 years ago

Hello!

This thread is closed. After further investigation I tried possible solutions to a bit different issues with ZT in linux. I finally managed to get a working zt0 interface. This thread helped by the the "fix": https://github.com/zerotier/ZeroTierOne/issues/809

Apparently the issue was a "rights issue"; adding the -U option as described down below fixed it.

`/lib/systemd/system/zerotier-one.service:

[Unit] Description=ZeroTier One After=network.target

[Service] ExecStart=/usr/sbin/zerotier-one -U Restart=always KillMode=process

[Install] `WantedBy=multi-user.target``

vamposdecampos commented 5 years ago

(on an openvz VPS) I've also had to chmod 777 /dev/net as well.

NeverBehave commented 4 years ago

Just a quick note if you google and find this issue: Don't forget to try rebooting

I have all settings correct (permission, etc.) but still encounter this problem, but it works after rebooting the machine.

diwu1989 commented 4 years ago

Please don't 777 the /dev/net these are safer alternatives:

chmod 755 /dev/net
chmod 666 /dev/net/tun
benhbell commented 4 years ago

Both of these also helped me with a node on OpenVz

rumym commented 2 years ago
chmod 666 /dev/net/tun

FWIW only 777 worked for me. I am logging in as root. chmod 777 /dev/net chmod 777 /dev/net/tun And it starts working!

lucas5-code commented 9 months ago

I also encountered this issue with mine. After uninstalling and reinstalling the latest version, it returned to normal

CevreMuhendisi commented 4 months ago

Thank you so much