Unable to ping

[ghahramani]

Hi I use Openwrt 23.05.2 and installed zerotier from the package manager, the version is 1.12.2

It seems it does not work as I cannot ping my network nodes, here is the info

root@MainRouter:~# zerotier-cli info
200 info XXXXXXXX 1.12.2 ONLINE
root@MainRouter:~# zerotier-cli listpeers
200 listpeers <ztaddr> <path> <latency> <version> <role>
200 listpeers XXXXXXXXX XX.209.XXX.104/21044;10711;10612 96 1.14.0 LEAF
200 listpeers XXXXXXXXX - -1 - PLANET
200 listpeers XXXXXXXXX XX.17.XX.155/9993;696;8973 39 - PLANET
200 listpeers XXXXXXXXX XX.XXX.98.98/9993;35756;125749 161 - PLANET
200 listpeers XXXXXXX - -1 - PLANET
root@MainRouter:~# zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
48d6023c46 1.14.0 LEAF      96 DIRECT   17191    17191    XX.XXX.XXX.104/21044
778cde7190 -      PLANET    -1 RELAY
cafe04eba9 -      PLANET    39 DIRECT   1644     29964    XX.XX.XX.155/9993
cafe9ccda7 -      PLANET   161 DIRECT   56747    146740   XX.XX.XX.98/9993
cafe9efeb9 -      PLANET    -1 RELAY
root@MainRouter:~# zerotier-cli listnetworks
200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
200 listnetworks XXXXXXXXXX navid_home XX:XX:XX:XX:XX:1b OK PRIVATE ztosinos7g fd48:XXXX:XXXX:d577:XXXX:932a:XXXX:7919/88,10.0.0.9/24

Here is my ping attempt

root@MainRouter:~# ping -c 1 10.4
PING 10.4 (10.0.0.4): 56 data bytes

--- 10.4 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@MainRouter:~# ping -c 1 10.9
PING 10.9 (10.0.0.9): 56 data bytes
64 bytes from 10.0.0.9: seq=0 ttl=64 time=0.358 ms

--- 10.9 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.358/0.358/0.358 ms
root@MainRouter:~# ping -c 1 10.2
PING 10.2 (10.0.0.2): 56 data bytes

--- 10.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

And finally, I noticed, the RX on the interface is 0

[ogarcia]

Have you approved the (router) node in Zerotier central? If yes, have you created any special rules? If you run an ip a the ztosinos7g interface gets ip?

On the other hand in the peer list I only see one leaf, are the rest of the nodes up (the 10.0.0.4, 10.0.0.9...)?

[ghahramani]

Thank you for responding quickly.

To answer your questions:

1. Yes, I approved it, all three devices are approved (authorized)
2. Yes I created the special rules to route all 192.168.0.0/23 to 10.0.0.9 which is the Openwrt router, the screenshot is provided
3. Yes, if I run an ip a I can see the IP 10.0.0.9 is assigned to the interface, screenshot is provided
4. Yes, all of them are up and 10.0.0.4 can ping 10.0.0.2 and 10.0.0.2 can ping 10.0.0.4

[ghahramani]

To extend the information, here are the firewall rules as well, also I think it is worth mentioning that the other nodes are using the latest version of the Zerotier client which is 1.14.0, and the only node that uses 1.12.2 is the OpenWrt as the repository is not updated to the latest version of it

[ghahramani]

Update:

The more I look into it, the more I get confused

I started a droplet from Digital Ocean to test it via a real Ubuntu server and I found very strange behavior.

Both Android (10.0.0.2) and Openwrt (10.0.0.9) can ping and connect (I started a server on the specific port) to Droplet (10.0.0.7), this is true for the droplet as well that can ping both Android and Openwrt

No the strange thing is, Android and Openwrt cannot ping each other.

Here is the ping from the droplet (10.0.0.7)

root@droplet:~# ping 10.2 -c 1
PING 10.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=209 ms

--- 10.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 208.686/208.686/208.686/0.000 ms
root@droplet:~# ping 10.9 -c 1
PING 10.9 (10.0.0.9) 56(84) bytes of data.
64 bytes from 10.0.0.9: icmp_seq=1 ttl=64 time=158 ms

--- 10.9 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 158.314/158.314/158.314/0.000 ms
root@droplet:~# ping 10.7 -c 1
PING 10.7 (10.0.0.7) 56(84) bytes of data.
64 bytes from 10.0.0.7: icmp_seq=1 ttl=64 time=0.047 ms

--- 10.7 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.047/0.047/0.047/0.000 ms

As you can see it can ping both Android (10.0.02) and Openwrt (10.0.0.9)

Now the ping from the openwrt

root@MainRouter:~# ping 10.2 -c 1
PING 10.2 (10.0.0.2): 56 data bytes

--- 10.2 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@MainRouter:~# ping 10.9 -c 1
PING 10.9 (10.0.0.9): 56 data bytes
64 bytes from 10.0.0.9: seq=0 ttl=64 time=0.219 ms

--- 10.9 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.219/0.219/0.219 ms
root@MainRouter:~# ping 10.7 -c 1
PING 10.7 (10.0.0.7): 56 data bytes
64 bytes from 10.0.0.7: seq=0 ttl=64 time=157.595 ms

--- 10.7 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 157.595/157.595/157.595 ms

As it is shown above, Openwrt (10.0.0.9) can ping Droplet (10.0.0.7) but cannot ping Android (10.0.0.2)

What is going on?

Note: After this test, I installed Netbird on all three devices and they could see each other without any problem, so the problem is really the Openwrt package that cannot work with Android

[ogarcia]

My advice is to test a little at a time instead of setting up everything at once. Delete all the configuration, create only the ZeroTier configuration and test the ping between devices before configuring the firewall for the VPN. Note that to ping your ZeroTier network you only need to have the network itself configured and UDP incoming traffic allowed on port 9993, which is done with the following configuration:

# Configure ZeroTier
uci set zerotier.openwrt_network=zerotier
uci add_list zerotier.openwrt_network.join='yournetworkid'
uci set zerotier.openwrt_network.enabled='1'

# Allow UDP packets to enter in ZeroTier
uci add firewall rule
uci set firewall.@rule[-1].name='Allow-ZeroTier-Inbound'
uci add_list firewall.@rule[-1].proto='udp'
uci set firewall.@rule[-1].src='*'
uci set firewall.@rule[-1].dest_port='9993'
uci set firewall.@rule[-1].target='ACCEPT'

# Commit changes
uci commit

Please note that ZeroTier uses UDP packets and that sometimes mobile networks cut this type of packets so, depending on your provider, ZeroTier may not be a solution that fits your needs.

[ghahramani]

I don't think UDP packets are the reason as I can ping my Android from the Digital Ocean droplet and I can ping my droplet from my Android.

I guess this is very specific to Zerotier in Openwrt, maybe something in the Openwrt is preventing the Zerotier to work but I am not sure where to start to debug

[ogarcia]

Have you tried what I have told you? Delete all the configuration and start from scratch configuring exclusively the ZeroTier network.

What I can assure you 100% is that either one of your providers is cutting some type of UDP communication or something is not properly configured. Having different versions of ZeroTier running is not a problem, I myself have 1.12.2 in OpenWrt and nodes with 1.14.0 (I even have a node with 1.8.4) and it works perfectly.

[ghahramani]

Yes, I did and I even completely reinstall the OpenWRT and installed the Zerotier on fresh Openwrt and still could not ping, to be fair, I could ping it for just 2 tries and then it became unreachable again on the first try on the newly installed OpenWrt

It is strange you can use it, what is the router and version of your Openwrt?

[ogarcia]

It is strange you can use it, what is the router and version of your Openwrt?

OpenWrt 23.05.4, the current stable release.

Does it work with WiFi on Android? Or does it always fail? You can try this other client which is said to be more complete.

[ghahramani]

Unfortunately, I have even tested that Android app, it does not work, I don't think the problem is the android, it is the openwrt as I can ping other peers from my Android, only the openwrt is not pingable/reachable.

One thing I forgot to mention is, that my DNS server is handled by the Adguardhome which is installed on the openwrt too, so Adguardhome forwards the DNS requests to Openwrt and Openwrt uses DNS over HTTPS plugin to serve the DNS but I don't think that can cause any problem, just worth it to mention

Also, I use PBR to route some specific ips to my miktorik router

[ogarcia]

Unfortunately, I have even tested that Android app, it does not work, I don't think the problem is the android, it is the openwrt as I can ping other peers from my Android, only the openwrt is not pingable/reachable.

But you had said that you can ping from and to OpenWrt from hosts other than Android. So your problem is not in OpenWrt, it is specifically in the communication between OpenWrt and Android, isn't it?

Also, I use PBR to route some specific ips to my miktorik router

My advice is to start from scratch with everything clean. You have something that specifically cuts off that communication between your OpenWrt router and your Android.

Another option is for you to set up a lab. Both Android and OpenWrt can be installed on virtual machines. You set up a virtualized environment and do some testing, you can even put a network sniffer in the middle to see what happens.

[ghahramani]

But you had said that you can ping from and to OpenWrt from hosts other than Android. So your problem is not in OpenWrt, it is specifically in the communication between OpenWrt and Android, isn't it?

But Android can ping other hosts too, so basically it is just between Openwrt and Android as each individual can ping other hosts without any problem

My advice is to start from scratch with everything clean. You have something that specifically cuts off that communication between your OpenWrt router and your Android.

I have to do it on my other router as this router has many configuration, I will do it.

Another option is for you to set up a lab. Both Android and OpenWrt can be installed on virtual machines. You set up a virtualized environment and do some testing, you can even put a network sniffer in the middle to see what happens.

Oh this is a good idea, I will try this tonight or tomorrow, thank you for the suggestion

One more thing, is there a way to debug what is going on here in general in Openwrt?

[ogarcia]

ne more thing, is there a way to debug what is going on here in general in Openwrt?

In OpenWrt you have the logread command that brings up all the logs in the terminal, but I don't know if it will tell you much in this case. Anyway, it is a Linux, you can install tcpdump and with that you can see everything that happens over the network. With tcpdump you can create a dump file that you then take to your machine and look at it with Wireshark. In the OpenWrt wiki there is an article that discusses how to do this.

[ogarcia]

A very important detail that I forgot to mention (it is impossible to remember everything) is that it is possible that the ping between Android and OpenWrt does not work because of how both OpenWrt and Android implement their respective firewalls. The best thing to do is the following:

1. We set up a ZeroTier network in the controller with the range 192.168.69.0/24.
2. We join the OpenWrt router and the Android to this network.
3. The OpenWrt router is assigned the IP 192.168.69.1 and the Android another one (for example 192.168.69.101).
4. Now we configure in the controller a route to reach the OpenWrt LAN. If the OpenWrt LAN is 192.168.1.0/24 then the route will have as destination 192.168.1.0/24 via 192.168.69.1 (the IP that we have assigned to the router in the ZeroTier network).
5. In the OpenWrt router if we do an ip a we will see all the interfaces, there will be one called zt******* (with the IP 192.168.69.1) we take note of the name to create an OpenWrt interface with the following commands:

   uci set network.ZeroTier=interface
   uci set network.ZeroTier.proto='none'
   uci set network.ZeroTier.device='zt******' # the name of ZeroTier interface
   uci commit

6. Configure the firewall to allow connections to and from the ZeroTier network (this is as is, it is not necessary to change anything).

   uci add firewall zone
   uci set firewall.@zone[-1].name='vpn'
   uci set firewall.@zone[-1].input='ACCEPT'
   uci set firewall.@zone[-1].output='ACCEPT'
   uci set firewall.@zone[-1].forward='ACCEPT'
   uci set firewall.@zone[-1].masq='1'
   uci add_list firewall.@zone[-1].network='ZeroTier'
   uci add firewall forwarding
   uci set firewall.@forwarding[-1].src='vpn'
   uci set firewall.@forwarding[-1].dest='lan'
   uci add firewall forwarding
   uci set firewall.@forwarding[-1].src='vpn'
   uci set firewall.@forwarding[-1].dest='wan'
   uci add firewall forwarding
   uci set firewall.@forwarding[-1].src='lan'
   uci set firewall.@forwarding[-1].dest='vpn'
   uci commit

7. In Android we now try to access the OpenWrt management web but through its LAN IP since by default it only listens on that interface. If the router has the IP 192.168.1.1 then we would have to access http://192.168.1.1
8. If that works, everything else should work.

[ogarcia]

@ghahramani has the last thing we have discussed worked for you?

[ghahramani]

@ogarcia Unfortunately, no. It did not work

Here is the config below

My phone could not connect to 192.168.1.1 which is my router :(

[ogarcia]

In the vpn zone configuration, you must allow forwarding to and from the lan. As I see in the screenshot you have enabled forwarding from lan to vpn but not the other way around.

On the other hand, you don't need the traffic rules that you see in this screenshot, that is already configured in the zone.

You also do not need to enable MSS Clamping.

The vpn zone configuration should look like this screenshot:

[ghahramani]

Wow, that works. Wonderful, thank you so much, I think we can close this ticket now but before that could you please reupload the first image so I can remove the unnecessary traffic rules as it does not show in your comment?

[ogarcia]

I have re-uploaded the image, I don't know why it wasn't showing, I was using the link from your comment. Weird stuff from GitHub. :smile: