search

openthread / ot-br-posix

OpenThread Border Router, a Thread border router for POSIX-based platforms.
https://openthread.io/
BSD 3-Clause "New" or "Revised" License
401 stars 226 forks source link

Cannot ping to an outside network from the OTBR #1793

Closed pesquivelm closed 1 year ago

pesquivelm commented 1 year ago

I am still very new to this so I appreciate if someone could please guide me. I have been learning about how to setup a thread network using a RockPi 4b with a nordic nrf52840 dongle for Border Router and nordic nrf52840 DK as child nodes. I have run the following commands to set up the border router:

for the Dongle

git clone --recursive https://github.com/openthread/ot-nrf528xx.git
cd ot-nrf528xx
./script/bootstrap
./script/build nrf52840 USB_trans -DOT_BOOTLOADER=USB

for the BR

git clone https://github.com/openthread/ot-br-posix
cd ot-br-posix
export NAT64=1 NAT64_SERVICE=openthread
NAT64=1 NAT64_SERVICE=openthread WEB_GUI=1 ./script/bootstrap
INFRA_IF_NAME=eth0 WEB_GUI=1 ./script/setup
sudo reboot

for the network

sudo ot-ctl
dataset init new
dataset commit active
ifconfig up
thread start

i have also tried nat64 enable. All of the above works fine. the problem im having is that i cant seem to ping to an external network. i get the following output no matter what ive tried

ping 8.8.8.8
1 packets transmitted, 0 packets received. Packet loss = 100.0%.
Done

net data looks like this

netdata show
Prefixes:
fda1:5c9a:3543:1::/64 paos low 4000
Routes:
fd36:e0fb:7450:410f::/64 s med 4000
Services:
44970 01 1e000500000e10 s 4000
44970 5d fd2bac79c35fadb98f22cc549791d0b9d11f s 4000

If theres any further info/data required ill gladly post it. Thanks in advance.

wgtdkp commented 1 year ago

It seems the NAT64 prefix (with /96 prefix length) is missing from the Routes list of your netdata show command. Did you forget to enable nat64 with command sudo ot-ctl nat64 enable?

Check https://openthread.io/codelabs/openthread-border-router-nat64#1 for detailed instructions

pesquivelm commented 1 year ago

@wgtdkp Thanks for your answer. Yes, sorry. I have run the process multiple times with different configurations and it seems that i posted the wrong output according to what i posted. I ran the commands posted above again and this is the output i got (its obviously different from before, but the commands i ran are the ones posted, including nat64 enable)

> netdata show
Prefixes:
fd6f:8470:e571:1::/64 paos low f800
Routes:
fd59:6604:9bb7:a5d3::/64 s med f800
fd6c:90db:955:9b78::/64 s med f800
fd6f:8470:e571:2:0:0::/96 sn low f800
fda5:9fdc:7c81:55ab::/64 s med f800
Services:
44970 01 63000500000e10 s f800
44970 5d fd051b798bbe5d1d490ef330c4a9fa3cd11f s f800
Done

the result for the ping command is still unsuccessful.

> ping 8.8.8.8
Pinging synthesized IPv6 address: fd6f:8470:e571:2:0:0:808:808
1 packets transmitted, 0 packets received. Packet loss = 100.0%.
Done
pesquivelm commented 1 year ago

I would also like to add that a "normal" ping directly from the rockpi (not using ot-ctl) does work without problems. so im not sure if it can be a firewall problem or something related

wgtdkp commented 1 year ago

so im not sure if it can be a firewall problem or something related

It's possible.

Would you follow below instructions to help diagnostic:

  1. Please Confirm eth0 is your infra link which provides internet access for your device. You are not connecting via Wi-Fi, right?
  2. May you share output of ifconfig, ip -6 route list table all and sudo ip6tables -L?
  3. Try capturing traffic on both the eth0 and wpan0 interfaces. To make sure the ping message has been successfully forwarded to your infra link
wgtdkp commented 1 year ago

@erjiaqing May know what's wrong here

pesquivelm commented 1 year ago

Yes. I am connected via ethernet, not wi-fi.

heres the ifconfig output

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.74  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::9327:3185:2f3:ae59  prefixlen 64  scopeid 0x20<link>
        inet6 fda5:9fdc:7c81:55ab:e6f8:9cc0:e75a:d4a5  prefixlen 64  scopeid 0x0                                                <global>
        ether c2:7b:6b:ce:22:56  txqueuelen 1000  (Ethernet)
        RX packets 36988  bytes 7669716 (7.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44422  bytes 12164289 (11.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 35

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 37  bytes 5269 (5.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 37  bytes 5269 (5.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

nat64: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 192.168.255.1  netmask 255.255.255.255  destination 192.168.255.1
        inet6 fe80::ff4a:d250:c250:32a  prefixlen 64  scopeid 0x20<link>
        inet6 fdaa:bb:1::2  prefixlen 128  scopeid 0x0<global>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500                                                  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5  bytes 340 (340.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1500
        inet 10.8.0.78  netmask 255.255.255.255  destination 10.8.0.77
        inet6 fe80::9143:e3b3:cc80:793b  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 100                                                  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4  bytes 304 (304.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlan0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether b0:02:47:93:8b:01  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wpan0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1280
        inet6 fe80::c066:1d3a:f897:b555  prefixlen 64  scopeid 0x20<link>
        inet6 fd05:1b79:8bbe:5d1d:0:ff:fe00:f800  prefixlen 64  scopeid 0x0<glob                                                al>
        inet6 fd05:1b79:8bbe:5d1d:0:ff:fe00:fc00  prefixlen 64  scopeid 0x0<glob                                                al>
        inet6 fd6f:8470:e571:1:51e4:78b1:c0cf:183f  prefixlen 64  scopeid 0x0<gl                                                obal>
        inet6 fd05:1b79:8bbe:5d1d:0:ff:fe00:fc38  prefixlen 64  scopeid 0x0<glob                                                al>
        inet6 fd05:1b79:8bbe:5d1d:0:ff:fe00:fc10  prefixlen 64  scopeid 0x0<glob                                                al>
        inet6 fd05:1b79:8bbe:5d1d:490e:f330:c4a9:fa3c  prefixlen 64  scopeid 0x0                                                <global>
        inet6 fd05:1b79:8bbe:5d1d:0:ff:fe00:fc11  prefixlen 64  scopeid 0x0<glob                                                al>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500                                                  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 10  bytes 1900 (1.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

here is the ip -6 route list table all

sudo ip -6 route list table all
::1 dev lo proto kernel metric 256 pref medium
2001:db8:1:ffff::/96 dev nat64 metric 1024 pref medium
fd05:1b79:8bbe:5d1d::/64 dev wpan0 proto kernel metric 256 pref medium
fd6f:8470:e571:1::/64 dev wpan0 proto kernel metric 256 pref medium
fda5:9fdc:7c81:55ab::/64 dev eth0 proto ra metric 100 pref medium
fdaa:bb:1::2 dev nat64 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 100 pref medium
fe80::/64 dev nat64 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium
fe80::/64 dev wpan0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast fd05:1b79:8bbe:5d1d:: dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:f800 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc00 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc10 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc11 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc38 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:490e:f330:c4a9:fa3c dev wpan0 table local proto kernel metric 0 pref medium
anycast fd6f:8470:e571:1:: dev wpan0 table local proto kernel metric 0 pref medium
local fd6f:8470:e571:1:51e4:78b1:c0cf:183f dev wpan0 table local proto kernel metric 0 pref medium
local fda5:9fdc:7c81:55ab:e6f8:9cc0:e75a:d4a5 dev eth0 table local proto kernel metric 0 pref medium
local fdaa:bb:1::2 dev nat64 table local proto kernel metric 0 pref medium
anycast fe80:: dev nat64 table local proto kernel metric 0 pref medium
anycast fe80:: dev tun0 table local proto kernel metric 0 pref medium
anycast fe80:: dev wpan0 table local proto kernel metric 0 pref medium
local fe80::9143:e3b3:cc80:793b dev tun0 table local proto kernel metric 0 pref medium
local fe80::9327:3185:2f3:ae59 dev eth0 table local proto kernel metric 0 pref medium
local fe80::c066:1d3a:f897:b555 dev wpan0 table local proto kernel metric 0 pref medium
local fe80::ff4a:d250:c250:32a dev nat64 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev nat64 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev tun0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wpan0 table local proto kernel metric 256 pref medium

and heres the sudo ip6tables -L

sudo ip6tables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
OTBR_FORWARD_INGRESS  all      anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain OTBR_FORWARD_INGRESS (1 references)
target     prot opt source               destination
DROP       all      anywhere             anywhere             PKTTYPE = unicast
DROP       all      anywhere             anywhere             match-set otbr-ingress-deny-src src
ACCEPT     all      anywhere             anywhere             match-set otbr-ingress-allow-dst dst
DROP       all      anywhere             anywhere             PKTTYPE = unicast
ACCEPT     all      anywhere             anywhere

for the last part (.3) i dont really know how to do it. I'll do a quick search and see what i can find about it. but in the meantime ill leave you with the previous results.

erjiaqing commented 1 year ago

Could you also check the status of IPv4 related rules?

pesquivelm commented 1 year ago

Ok. for the forwarding I think it is enabled. here is the output

cat /proc/sys/net/ipv4/conf/all/forwarding
1

IP tables outputs this. Im not sure if i have NAT44 enabled.

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
erjiaqing commented 1 year ago

Ok. for the forwarding I think it is enabled. here is the output

cat /proc/sys/net/ipv4/conf/all/forwarding
1

Seems fine

IP tables outputs this. Im not sure if i have NAT44 enabled.

sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Ok, the forwarding chain is not blocked.

Could you also check the nat table (or iptables -t nat -L -v) (Sorry, I should mention this in previous comment).

pesquivelm commented 1 year ago

No worries! im appreciating all the help im getting. heres what iptables -t nat -L -v shows

 sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  any    any     192.168.255.0/24     anywhere
    0     0 MASQUERADE  all  --  any    any     192.168.255.0/24     anywhere

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
erjiaqing commented 1 year ago

Seems there are no issues with the iptables rules.

However, it is quite strange that the counters in the iptables are 0. Did you rebooted the devices recently? Or could you try to ping an IPv4 address from the child and try to see if the packet is counted?

Also, could you check the status and the counters on the BR by running the following commands on the BR?

$ ot-ctl nat64 state
$ ot-ctl nat64 counters
$ ot-ctl nat64 mappings

OpenThread will configure the IPv4 route for NAT64 so usually it is not an issue, however, could you also check if it is congured as expected by ip route list table all

pesquivelm commented 1 year ago

yes i did turn off the device. also, what by child do you mean a node that is not the Border Router? Here are the results for the commands you posted

sudo ot-ctl nat64 state
PrefixManager: Active
Translator: Active
Done
sudo ot-ctl nat64 counters
|               | 4 to 6                  | 6 to 4                  |
+---------------+-------------------------+-------------------------+
| Protocol      | Pkts     | Bytes        | Pkts     | Bytes        |
+---------------+----------+--------------+----------+--------------+
|         Total |        0 |            0 |        0 |            0 |
|           TCP |        0 |            0 |        0 |            0 |
|           UDP |        0 |            0 |        0 |            0 |
|          ICMP |        0 |            0 |        0 |            0 |
| Errors        | Pkts                    | Pkts                    |
+---------------+-------------------------+-------------------------+
|       Unknown |                       0 |                       0 |
|   Illegal Pkt |                       0 |                       0 |
|   Unsup Proto |                       0 |                       0 |
|    No Mapping |                       0 |                       0 |
Done
sudo ot-ctl nat64 mappings
|                  | Address                                                     |        | 4 to 6                  | 6 to 4                  |
+------------------+-------------------------------------------------------------+--------+-------------------------+-------------------------+
| ID               | IPv6                                     | IPv4             | Expiry | Pkts     | Bytes        | Pkts     | Bytes        |
+------------------+------------------------------------------+------------------+--------+----------+--------------+----------+--------------+
Done

heres for ip route list table all

 ip route list table all
default via 192.168.1.254 dev eth0 proto dhcp metric 100
default via 192.168.1.254 dev eth0 proto dhcp src 192.168.1.74 metric 202
10.8.0.0/24 via 10.8.0.77 dev tun0
10.8.0.77 dev tun0 proto kernel scope link src 10.8.0.78
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.74 metric 100
192.168.1.0/24 dev eth0 proto dhcp scope link src 192.168.1.74 metric 202
192.168.255.0/24 dev nat64 scope link
192.168.255.0/24 dev wpan0 metric 100
local 10.8.0.78 dev tun0 table local proto kernel scope host src 10.8.0.78
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev eth0 table local proto kernel scope link src 192.168.1.74
local 192.168.1.74 dev eth0 table local proto kernel scope host src 192.168.1.74
broadcast 192.168.1.255 dev eth0 table local proto kernel scope link src 192.168.1.74
local 192.168.255.1 dev nat64 table local proto kernel scope host src 192.168.255.1
::1 dev lo proto kernel metric 256 pref medium
2001:db8:1:ffff::/96 dev nat64 metric 1024 pref medium
fd05:1b79:8bbe:5d1d::/64 dev wpan0 proto kernel metric 256 pref medium
fd6f:8470:e571:1::/64 dev wpan0 proto kernel metric 256 pref medium
fda5:9fdc:7c81:55ab::/64 dev eth0 proto ra metric 100 pref medium
fdaa:bb:1::2 dev nat64 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 100 pref medium
fe80::/64 dev nat64 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium
fe80::/64 dev wpan0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast fd05:1b79:8bbe:5d1d:: dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:f800 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc00 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc10 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc11 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:0:ff:fe00:fc38 dev wpan0 table local proto kernel metric 0 pref medium
local fd05:1b79:8bbe:5d1d:490e:f330:c4a9:fa3c dev wpan0 table local proto kernel metric 0 pref medium
anycast fd6f:8470:e571:1:: dev wpan0 table local proto kernel metric 0 pref medium
local fd6f:8470:e571:1:51e4:78b1:c0cf:183f dev wpan0 table local proto kernel metric 0 pref medium
local fda5:9fdc:7c81:55ab:e6f8:9cc0:e75a:d4a5 dev eth0 table local proto kernel metric 0 pref medium
local fdaa:bb:1::2 dev nat64 table local proto kernel metric 0 pref medium
anycast fe80:: dev nat64 table local proto kernel metric 0 pref medium
anycast fe80:: dev tun0 table local proto kernel metric 0 pref medium
anycast fe80:: dev wpan0 table local proto kernel metric 0 pref medium
local fe80::7a5c:2a12:10f0:6521 dev nat64 table local proto kernel metric 0 pref medium
local fe80::9327:3185:2f3:ae59 dev eth0 table local proto kernel metric 0 pref medium
local fe80::c066:1d3a:f897:b555 dev wpan0 table local proto kernel metric 0 pref medium
local fe80::d73e:6722:fefc:9969 dev tun0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev nat64 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev tun0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wpan0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
erjiaqing commented 1 year ago

From the route, seems tayga was installed before

You can try to disable the tayga (by systemctl disable tayga or uninstall it directly) and try if NAT64 works.

pesquivelm commented 1 year ago

Yes! that seems to have worked! i guess tayga was causing conflicts with NAT64, as you say. i disabled and stopped the service and now ot-ctl ping works!!

systemctl disable tayga
systemctl stop tayga
systemctl status tayga
● tayga.service - LSB: userspace NAT64
   Loaded: loaded (/etc/init.d/tayga; generated)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

 sudo ot-ctl

>
> ping 8.8.8.8
Pinging synthesized IPv6 address: fd6f:8470:e571:2:0:0:808:808
16 bytes from fd6f:8470:e571:2:0:0:808:808: icmp_seq=2 hlim=116 time=60ms
1 packets transmitted, 1 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 60/60.0/60 ms.
Done

Thank you very much for your help!! i really can't thank you enough!