ESP32 Router: Unable to ping RIOT node from Linux

srigirish commented 1 year ago

Description

I have an ESP32 setup as a router using gnrc_ipv6_router_default and a Nucleo-F767ZI as a RIOT node connected to the ESP32 via SLIP using UART. I have modified the gnrc_border_router example and the SUIT update example with the help of @benpicco (see: https://forum.riot-os.org/t/border-router-ota-using-wifi/3895/40) to convert the border router into a WiFi <-> SLIP router. I am successfully able to ping the ESP32 and the Nucleo back and forth, and I am able to ping the ESP32 through the WiFi interface of my Linux PC. I am unable to ping the RIOT node from my Linux PC, there is no response from the node.

I'm not sure if this issue has been addressed before but I have not seen many issues here which had a WiFi to SLIP router, and I don't know if I need to do anything else to make this work. But I've been struggling with this for a couple of days before I decided to post it here.

Steps to reproduce the issue

Flash gnrc_border_router to ESP32

BOARD=esp32-wroom-32 UPLINK=wifi PREFIX_CONF=static make -C examples/gnrc_border_router/ flash term

Connect ESP32 to the same WiFi network as the Linux PC
Flash suit_update to Nucleo-F767ZI USE_ETHOS=0 BOARD=nucleo-f767zi PROGRAMMER=cpy2remed make -C examples/suit_update/ term
Connect ESP32 to the Nucleo board via UART (SLIP)
Add route to Nucleo board via ESP32 from Linux sudo ip route add 2001:db8::/64 via fe80::a6cf:12ff:fe9a:2dc8 dev wlp0s20f3

Here wlp0s20f3 is the name of the WiFi interaface of the Linux PC

Expected results

Nucleo board responds to the ping from Linux

Actual results

Ping ESP32 from Linux (Success)

Ping Nucleo from Linux (Failure)

Versions

RIOT Version: 2023.04-devel-420-g96a7d

srigirish commented 1 year ago

Can I get some help on this? I have been wracking my brain for two weeks but I cannot seem to figure out the problem here.

benpicco commented 1 year ago

I can't reproduce your issue.

There is indeed a bug when you use gnrc_sixlowpan_border_router_default on the ~~border~~ router, but that can be mitigated by using gnrc_ipv6_router_default instead or applying #19900. Since your downstream node did obtain the prefix, it looks like you already did that.

Maybe if I post my setup that helps you to solve your issue:

The ESP32 is connected to a WiFi network that is bridged to the LAN that my PC is on. It's ethernet interface is:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 14:da:e9:4c:6b:2e brd ff:ff:ff:ff:ff:ff
    altname enp0s25
    inet 192.168.42.105/24 brd 192.168.42.255 scope global dynamic noprefixroute eno1
       valid_lft 854815sec preferred_lft 854815sec
    inet6 2001:9e8:1401:b000:5153:9266:9ed6:34f9/64 scope global temporary dynamic 
       valid_lft 6731sec preferred_lft 3131sec
    inet6 2001:9e8:1401:b000:2b35:8f23:4da:60f6/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 6731sec preferred_lft 3131sec
    inet6 fe80::fe08:c033:940c:7e79/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Now we need the link-local address of my machine to set it to be the default router of the network (because we want the ESP32 to forward packets from the downstream network to it): IPV6_DEFAULT_ROUTER=fe80::fe08:c033:940c:7e79

You already have the changes to enable SLIP and disable esp_now in your Makefile, just to compare, the diff is

--- a/examples/gnrc_border_router/Makefile.board.dep
+++ b/examples/gnrc_border_router/Makefile.board.dep
@@ -8,8 +8,11 @@ ifeq (,$(filter native,$(BOARD)))
     ifneq (,$(filter esp32 esp8266,$(CPU)))
       USEMODULE += esp_wifi
       ifneq (ble, $(DOWNLINK))
-        USEMODULE += esp_now
+#        USEMODULE += esp_now
       endif
+      USEMODULE += slipdev_l2addr
+      CFLAGS += -DCONFIG_GNRC_IPV6_STATIC_ADDR_UPSTREAM=8
+      CFLAGS += -DCONFIG_GNRC_IPV6_STATIC_ADDR_DOWNSTREAM=9
     else
       $(error Only esp32 and esp8266 are currently supported)
     endif

To flash this I use

make BOARD=esp32-wroom-32 UPLINK=wifi PREFIX_CONF=static IPV6_PREFIX=2001:db8::/64 IPV6_DEFAULT_ROUTER=fe80::fe08:c033:940c:7e79 flash term

This gives on the ESP32 router:

2023-08-24 00:38:03,378 # Iface  8  HWaddr: 3C:71:BF:9E:13:FC  Channel: 6  Link: up 
2023-08-24 00:38:03,382 #           L2-PDU:1500  MTU:1500  HL:64  RTR  
2023-08-24 00:38:03,385 #           Source address length: 6
2023-08-24 00:38:03,388 #           Link type: wireless
2023-08-24 00:38:03,393 #           inet6 addr: 2001:db8:1::1  scope: global  VAL
2023-08-24 00:38:03,399 #           inet6 addr: fe80::3e71:bfff:fe9e:13fc  scope: link  VAL
2023-08-24 00:38:03,402 #           inet6 group: ff02::2
2023-08-24 00:38:03,404 #           inet6 group: ff02::1
2023-08-24 00:38:03,407 #           inet6 group: ff02::1:ff00:1
2023-08-24 00:38:03,411 #           inet6 group: ff02::1:ff9e:13fc
2023-08-24 00:38:03,412 #           
2023-08-24 00:38:03,413 # Iface  9 
2023-08-24 00:38:03,417 #           Long HWaddr: A6:C9:B8:3A:F6:29:C0:D9 
2023-08-24 00:38:03,420 #           MTU:65535  HL:64  RTR  
2023-08-24 00:38:03,422 #           RTR_ADV  
2023-08-24 00:38:03,424 #           Link type: wired
2023-08-24 00:38:03,430 #           inet6 addr: fe80::a4c9:b83a:f629:c0d9  scope: link  VAL
2023-08-24 00:38:03,436 #           inet6 addr: 2001:db8::a4c9:b83a:f629:c0d9  scope: global  VAL
2023-08-24 00:38:03,438 #           inet6 group: ff02::2
2023-08-24 00:38:03,441 #           inet6 group: ff02::1
2023-08-24 00:38:03,445 #           inet6 group: ff02::1:ff29:c0d9

2023-08-24 00:38:18,038 # nib prefix
2023-08-24 00:38:18,042 # 2001:db8:1::1/128 dev #8 
2023-08-24 00:38:18,044 # 2001:db8::/64 dev #9

2023-08-24 00:41:05,499 # nib route
2023-08-24 00:41:05,504 # 2001:db8:1::1/128 dev #8
2023-08-24 00:41:05,506 # 2001:db8::/64 dev #9
2023-08-24 00:41:05,510 # default* via fe80::fe08:c033:940c:7e79 dev #8

I can now add the route to the downstream network (2001:db8::/64) via the upstream interface of the ESP32 (fe80::3e71:bfff:fe9e:13fc):

sudo ip route add 2001:db8::/64 via fe80::3e71:bfff:fe9e:13fc dev eno1

The downstream node should have obtained the prefix from the ESP32 via SLIP by now too:

2023-08-24 00:39:37,573 # Iface  5 
2023-08-24 00:39:37,577 #           Long HWaddr: EA:64:83:F2:1B:AA:05:B1 
2023-08-24 00:39:37,581 #           MTU:65535  HL:64  
2023-08-24 00:39:37,584 #           Link type: wired
2023-08-24 00:39:37,590 #           inet6 addr: fe80::e864:83f2:1baa:5b1  scope: link  VAL
2023-08-24 00:39:37,596 #           inet6 addr: 2001:db8::e864:83f2:1baa:5b1  scope: global  VAL
2023-08-24 00:39:37,599 #           inet6 group: ff02::1
2023-08-24 00:39:37,603 #           inet6 group: ff02::1:ffaa:5b1

and with the route in place I can ping it from Linux:

% ping -c3 2001:db8::e864:83f2:1baa:5b1
PING 2001:db8::e864:83f2:1baa:5b1(2001:db8::e864:83f2:1baa:5b1) 56 data bytes
64 bytes from 2001:db8::e864:83f2:1baa:5b1: icmp_seq=1 ttl=63 time=83.7 ms
64 bytes from 2001:db8::e864:83f2:1baa:5b1: icmp_seq=2 ttl=63 time=107 ms
64 bytes from 2001:db8::e864:83f2:1baa:5b1: icmp_seq=3 ttl=63 time=128 ms

--- 2001:db8::e864:83f2:1baa:5b1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 83.720/106.221/128.004/18.086 ms

srigirish commented 1 year ago

Thank you @benpicco for posting your setup in such detail. The only difference I can find between my setup and yours is that both the upstream interface in your ESP32 and your PC's ethernet interface have a global address, both of which are missing in mine. Are these necessary? Are there any extra steps which I'm missing to replicate your setup?

Here are my outputs: WiFi interface of my PC:

ESP32 outputs:

Nucleo output:

benpicco commented 1 year ago

When you do a

udp server start 1234

on the nucleo board and then do a

echo Hello | nc -6 -u 2001:db8::ac97:d831:a613:bd6e

on your PC, does that arrive on the nucleo board?

What's the output of

ip -6 route

on your PC?

srigirish commented 1 year ago

The output of ip -6 route is

::1 dev lo proto kernel metric 256 pref medium
2001:db8::/64 via fe80::a6cf:12ff:fe9a:2dc8 dev wlp0s20f3 metric 1024 pref medium
fe80::/64 dev wlp0s20f3 proto kernel metric 1024 pref medium

The command echo Hello | nc -6 -u 2001:db8::ac97:d831:a613:bd6e gives an error nc: missing port number When I run echo Hello | nc -6 -u 2001:db8::ac97:d831:a613:bd6e 1234 it doesn't print anything, and nothing happens in the Nucleo as well.

Nucleo:

PC: It is just stuck there.

EDIT: Adding an extra -v flag provided one line of output. Still stuck after this line is printed

benpicco commented 1 year ago

RIOT Version: 2023.04-devel-420-g96a7d

hm this seems a bit old, have you tried with the latest master?

srigirish commented 1 year ago

I've just tried with the latest master and the ping still does not work. The only difference is that the ifconfig and the nib outputs from the ESP32 router look similar to yours.

ifconfig
2023-08-24 16:48:19,706 # ifconfig
2023-08-24 16:48:19,711 # Iface  7  HWaddr: A4:CF:12:9A:2D:C8  Channel: 1  Link: up 
2023-08-24 16:48:19,715 #           L2-PDU:1500  MTU:1500  HL:64  RTR  
2023-08-24 16:48:19,718 #           Source address length: 6
2023-08-24 16:48:19,721 #           Link type: wireless
2023-08-24 16:48:19,726 #           inet6 addr: 2001:db8:1::1  scope: global  VAL
2023-08-24 16:48:19,732 #           inet6 addr: fe80::a6cf:12ff:fe9a:2dc8  scope: link  VAL
2023-08-24 16:48:19,734 #           inet6 group: ff02::2
2023-08-24 16:48:19,737 #           inet6 group: ff02::1
2023-08-24 16:48:19,740 #           inet6 group: ff02::1:ff00:1
2023-08-24 16:48:19,744 #           inet6 group: ff02::1:ff9a:2dc8
2023-08-24 16:48:19,745 #           
2023-08-24 16:48:19,746 # Iface  8 
2023-08-24 16:48:19,750 #           Long HWaddr: 66:11:DA:1D:BE:FC:7E:AF 
2023-08-24 16:48:19,753 #           MTU:65535  HL:64  RTR  
2023-08-24 16:48:19,754 #           RTR_ADV  
2023-08-24 16:48:19,757 #           Link type: wired
2023-08-24 16:48:19,762 #           inet6 addr: fe80::6411:da1d:befc:7eaf  scope: link  VAL
2023-08-24 16:48:19,769 #           inet6 addr: 2001:db8::6411:da1d:befc:7eaf  scope: global  VAL
2023-08-24 16:48:19,771 #           inet6 group: ff02::2
2023-08-24 16:48:19,774 #           inet6 group: ff02::1
2023-08-24 16:48:19,778 #           inet6 group: ff02::1:fffc:7eaf
2023-08-24 16:48:19,779 #           
> nib route
2023-08-24 16:49:06,537 # nib route
2023-08-24 16:49:06,541 # 2001:db8:1::1/128 dev #7
2023-08-24 16:49:06,543 # 2001:db8::/64 dev #8
2023-08-24 16:49:06,547 # default* via fe80::8f58:edd8:53d8:534 dev #7
> nib prefix
2023-08-24 16:57:35,397 # nib prefix
2023-08-24 16:57:35,401 # 2001:db8:1::1/128 dev #7 
2023-08-24 16:57:35,403 # 2001:db8::/64 dev #8

benpicco commented 1 year ago

What's the output on the nucleo?

srigirish commented 1 year ago

Nucleo output:

> ifconfig
2023-08-24 18:21:04,464 # ifconfig
2023-08-24 18:21:04,468 # Iface  4  HWaddr: EA:C8:61:14:DB:02  Link: down 
2023-08-24 18:21:04,475 #           L2-PDU:1500  MTU:1500  HL:64  Source address length: 6
2023-08-24 18:21:04,477 #           Link type: wired
2023-08-24 18:21:04,483 #           inet6 addr: fe80::e8c8:61ff:fe14:db02  scope: link  VAL
2023-08-24 18:21:04,486 #           inet6 group: ff02::1
2023-08-24 18:21:04,489 #           inet6 group: ff02::1:ff14:db02
2023-08-24 18:21:04,490 #           
2023-08-24 18:21:04,491 # Iface  5 
2023-08-24 18:21:04,495 #           Long HWaddr: AE:97:D8:31:A6:13:BD:6E 
2023-08-24 18:21:04,498 #           MTU:65535  HL:64  
2023-08-24 18:21:04,501 #           Link type: wired
2023-08-24 18:21:04,506 #           inet6 addr: fe80::ac97:d831:a613:bd6e  scope: link  VAL
2023-08-24 18:21:04,513 #           inet6 addr: 2001:db8::ac97:d831:a613:bd6e  scope: global  VAL
2023-08-24 18:21:04,516 #           inet6 group: ff02::1
2023-08-24 18:21:04,519 #           inet6 group: ff02::1:ff13:bd6e
2023-08-24 18:21:04,520 #

srigirish commented 1 year ago

I have tried the same process at a different location using a different router and it worked. I can now see the ping reply.

ping 2001:db8::ac97:d831:a613:bd6e
PING 2001:db8::ac97:d831:a613:bd6e(2001:db8::ac97:d831:a613:bd6e) 56 data bytes
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=1 ttl=63 time=225 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=2 ttl=63 time=224 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=3 ttl=63 time=44.2 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=4 ttl=63 time=75.8 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=5 ttl=63 time=103 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=6 ttl=63 time=135 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=7 ttl=63 time=158 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=8 ttl=63 time=180 ms
64 bytes from 2001:db8::ac97:d831:a613:bd6e: icmp_seq=9 ttl=63 time=203 ms
^C
--- 2001:db8::ac97:d831:a613:bd6e ping statistics ---
9 packets transmitted, 9 received, 0% packet loss, time 8009ms
rtt min/avg/max/mdev = 44.221/149.870/225.095/61.405 ms

I guess IPV6 is turned off by default / not available on my home network, that's why it did not work previously. @benpicco Thanks for your help!

RIOT-OS / RIOT