Networkbridge with Raspberry Pi is not working

[MIPMHannes]

As you can see above I am trying bridge the RasPi Wlan0 interface with ESP32 ethap0 Interface. Sadly it is not quite working as expected. The Problem is that a client connect to ethap0 can not ping another client connected with wlan0. The bridge configuration seems to be alright since I can put another interface e.g eth0 into the bridge and it is working. So I figured this might be a bug with the esp-hosted firmware. I am using ESP-32-WROOM with the firmware 0.4. RasPi runs the 0.4 drivers and kernel version 5.15.61-v7l+. Strange thing is I can ping the ip address of the bridge Interface from the clients but I can not ping anything behind that bridge. Are you able to give me some hints? Thanks!

[mantriyogesh]

Thanks a lot for the diagram and the explanation of the issue.

Just to understand, in your scenario, your packets from the stations connected to softap (ethap0) are able to ping the RPI's IP address of ethap0 network interface?

If yes, You might have to just add additional routing rules at Raspberry Pi side switch packets from/to ethap0 network to/from eth0/wlan0 network.

APIs that may interest you, https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L197 and, https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L386

[MIPMHannes]

Thanks for your immediate reply! Since we use a network bridge ethap0 has no ip. The bridge interface br0 which ethap0 is member of has a ip address. Clients can ping the bridge interface br0. Can you clarify what you mean with

might have to just add additional routing rules at Raspberry Pi side switch packets from/to ethap0 network to/from eth0/wlan0 network.

APIs that may interest you,

https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L197

and, https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L386

and why the code snippets might interest me.

[mantriyogesh]

These are the RX/TX hooks for any trafffic relating to softap traffic (ethap0). You can have control of dropping any packets etc(in case you want to at ESP level)

The IP counterpart (routing/ifconfig/dhcp client,server) is actually out of scope for ESP-Hosted. But as you say, softap clients are able to ping to br0, that is great.

You might have to work on the routing on Linux side. What I perceive from discussion, ethap0 is working individually and wlan0 is working individually. Now only question remains is, packets forwarding from 'one' network subnet to 'another'. IMO, routing rules will be useful here. Anything else on your mind?

Please do enlighten me if I am missing some part from ESP-Hosted which is not working for you.

[MIPMHannes]

Okay I need to look further into that and figure out if this helps my case. I am trying to avoid routing anything in this case because it will produce too much overhead. So I am trying to do anything in layer 2. Usually this approach works quite well but not with eth0ap. I suggest arp is the mischief here. Down below you see a screenshot of tcpdump on RasPi interface br0. Is eth0ap not able handle with arp requests? Is this in the scope of this project? Thanks for your help!

[mantriyogesh]

Only filtering is done at this place for any RX packets: https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L210-L214 You might want comment this and try.

Other than above filtering, all the packets (including arping) are treated same. At L2 layer, we do not maintain any context. It should be transparent. Although, we might not have any control over arptable maintained at Raspberry Pi.

In case you want to handle something at L2, wlan_ap_rx_callback() or esp_wifi_internal_tx() could be used. I will further check if anything more can be possible, but But Raspberry Pi's network understanding would not be available as such at this (ESP) place.

[MIPMHannes]

Only filtering is done at this place for any RX packets:

https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L210-L214

I have tried this. I edited esp-hosted/esp/esp_driver/network_adapter/main/app_main.c and now it looks like this: After that I recompiled the drivers with /opt/esp-hosted/host/linux/host_control/rpi_init.sh spi. Tcpdump still looks like outlined above therefore ping does not work either. Did we do something wrong? Thanks (:

[mantriyogesh]

We can do in steps. 1) Get the ping working from station connected to softap works or not

• 1.1) Static IP
  • start softap, connect a station (please make sure any firewalls are disabled or iptable and routes are flushed at this connecting station)
  • First try with static IP (same network subnet for espap0 on RPi and station connected to softap)
  • This ping must work
• 1.2) DHCP
  • https://github.com/espressif/esp-hosted/blob/master/esp_hosted_fg/docs/common/c_demo.md#some-points-to-note -> 'SoftAP start' -> setup dnsmasq

Note: Ping anyway will make sure the arp entries are updated. Let us cleaned the manual arp entries (if any added) before trying step (1) 2) Get the basic routing working on your Raspberry Pi to forward packets 3) Any optimization you want to continue ahead

@MIPMHannes, These all things are little out of scope for ESP-Hosted. But anyway, we can try something together.

Before the hand, it will be better if you can tell in stepwise manner, how you are setting wrt network. For example, station to softap wifi is setup, what IPs you use, etc.

[MIPMHannes]

1.1 and 1.2 are already working. I am able to ping from laptop1 to br0. As stated before we are not trying to implement routing but want to bridge the interfaces ethap0 and wlan0 to create a broadcast domain. As you can see in the sketch everything works fine while using eth0 but it does not work with ethap0. Are you aware of any limitations when putting ethap0 in a bridge (br0)? I think it is a problem with esp-hosted because everything outside works. Thanks for your help!

[mantriyogesh]

Oh! Understood the problem.

Consider the point that, ethap0 is virtual interface created. If any manually created virtual interface can be added in br0, espap0 will also should work in same way.

However, there might be good amount of differences in between real ethernet interface and virtual interface. For example, We have implemented minimally required interfaces as part of struct net_device_ops https://github.com/espressif/esp-hosted/blob/f1d32abd183d6e4dd0cfb118b7a91faf4c1b674b/esp_hosted_fg/host/linux/host_driver/esp32/main.c#L93-L101

We might have to then compare these things that point of view. Do you get any logs/errors when you are creating br0 with ethap0?

This should definitely be do-able. But might take some time to understand & implement the expectations for br0.

[MIPMHannes]

I am not sure if I can give you any meaningful logs since I have invested countless hours in debugging and could not find any helpful logs. It would be great if this issue will be fixed. I think there are a lot of use cases where esp-hosted working flawlessly on Layer2 may came in handy. Thank you so much for your afford and time! @mantriyogesh please feel free to contact me if you need some further information :)

[MIPMHannes]

Are this logs helpful? First is systemd log while repopulating the interfaces second is dmesg log while restarting systemd-networkd. systemd_networkd.log dmesg2.log

[mantriyogesh]

As such, There is no visible issues in the logs. Can you please also share how did you create the bridge (to mimic the steps) & config if any used for creation of bridge?

[MIPMHannes]

Yes of course.

Creating the bridge

Create file /etc/sytemd/netword/br0.netdev with following content:

[NetDev]
Name=br0  
Kind=bridge

Then create file /etc/systemd/network/ethap0.network with following content:

[Match]
Name=ethap0
[Network]
Bridge=br0

Now create file /etc/systemd/network/br0.network with following content:

[Match]
Name=br0

[Network]
DHCP=yes

br0 does not have a ip address yet. You can assign one by adding following lines to your dhcpcd.conf

interface br0
static ip_address=192.168.4.253/24

You can restart systemd-networkd after that. The bridge is now configured. However wlan0 is not a member and not configured as AP. I will do both in the next step.

Configuring wlan0

I use hostapd in order to configure wlan0 as an AP. You need to install it first (apt install hostapd) Create a file /etc/hostapd/hostapd.conf with following content:

# Schnittstelle und Treiber
interface=wlan0
bridge=br0 # this adds wlan0 to bridge

# WLAN-Konfiguration
ssid=WAP
hw_mode=g
channel=1
macaddr_acl=0
ignore_broadcast_ssid=0

# WLAN-Verschlüsselung
auth_algs=1
wpa=2
wpa_passphrase=passwort
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

Hint: you may want to add denyinterfaces wlan0 ethap0 to /etc/dhcpd.conf otherwise the interfaces might get ip addresses.

Please reach out if something is unclear with the setup. Thanks for your help!

[mantriyogesh]

Thanks, @MIPMHannes, for the set-up explanation.

Give me some time, We will pick this issue once current priority items are done. Please bear with us by the time.

[MIPMHannes]

Can you give an estimate when this will be fixed?

[mantriyogesh]

Hello @MIPMHannes This is interesting to fix, but the problem is that we are working on some other priority things, because of which this is getting delayed.

[Philipansari]

Any news on this topic?

I have basically the same problem. I added the ethap0 interface to an existing bridge and the stations are not able to communicate to other interfaces inside the bridge.

One interesting part to mention is that the stations itself are only able to ping the softap (ethap0 interface) if the bridge has the mac-address of the ethap0 interface. If the bridge has a different mac-address than the ethap0 interface then the ARP-reply problem arises also between stations <---> softap (at least in my case).

[mantriyogesh]

1. The Mac address of the AP interface, on ESP side, should be equal to the mac address of ethap0 on Linux virtual network interface.
2. You can change the MAC address of ESP if you wish to, using wifi_set_mac() API provided with mode = WIFI_MODE_AP. What happens if you keep the ESP's mac equal to bridge mac, which is different than that of espap0 on Linux?
3. Can you try commenting the following code: https://github.com/espressif/esp-hosted/blob/ce3c50a33fa4bc562a1b6cbcee292c1ae0b0a404/esp_hosted_fg/esp/esp_driver/network_adapter/main/app_main.c#L225-L229

Bridge cases are never tried and tested yet, btw.

[Philipansari]

Thanks for the fast reply :)

I commented the code you mentioned and indeed it start working. The ESP-driver was probably ignoring packages which were meant for the bridge interface, but not directly addressed to the ethap0 interface. Great!

Now, why is the filter there. Which consequences does it have removing it?

[mantriyogesh]

It was added to drop unnecessary traffic which is non intended (broadcast, multicast) for softap. Only allow unicast itself. Anyway in your case you use the interface, so makes sense to comment out this code.

We are thinking to by default disable it and only enable if user intend to..