ESP32 MQTT traffic not being detected via Wireshark (IDFGH-10056)

[krupis]

Answers checklist.

• [X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• [X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• [X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hello. I am not sure whether this is correct place for creating an esp-idf ticket, but I thought I will give it a try. Perhaps someone else had simmilar issues.

I use ESP32-Ethernet-Kit V1.2 I have compiled and flashed the esp-idf mqtt 5 example: esp\esp-idf\examples\protocols\mqtt5

The only thing I have change in the menuconfig is instead of wifi connection I use ethernet connection. From the ESP32 logs, I can see that everyting seems to be fine.

I (4540) example_common: - IPv6 address: fe80:0000:0000:0000:c6dd:57ff:fe5f:290b, type: ESP_IP6_ADDR_IS_LINK_LOCAL
I (4560) MQTT5_EXAMPLE: Other event id:7
I (4790) MQTT5_EXAMPLE: MQTT_EVENT_CONNECTED
I (4790) MQTT5_EXAMPLE: sent publish successful, msg_id=16070
I (4800) MQTT5_EXAMPLE: sent subscribe successful, msg_id=47982
I (4800) MQTT5_EXAMPLE: sent subscribe successful, msg_id=41654
I (4810) MQTT5_EXAMPLE: sent unsubscribe successful, msg_id=61013
I (4910) mqtt5_client: MQTT_MSG_TYPE_PUBACK return code is -1
I (4910) MQTT5_EXAMPLE: MQTT_EVENT_PUBLISHED, msg_id=16070
I (5170) mqtt5_client: MQTT_MSG_TYPE_SUBACK return code is 0
I (5170) MQTT5_EXAMPLE: MQTT_EVENT_SUBSCRIBED, msg_id=47982
I (5170) MQTT5_EXAMPLE: sent publish successful, msg_id=0
I (5180) mqtt5_client: MQTT_MSG_TYPE_SUBACK return code is 2
I (5180) MQTT5_EXAMPLE: MQTT_EVENT_SUBSCRIBED, msg_id=41654
I (5190) MQTT5_EXAMPLE: sent publish successful, msg_id=0
I (5200) MQTT5_EXAMPLE: MQTT_EVENT_DATA
I (5200) MQTT5_EXAMPLE: key is board, value is esp32
I (5200) MQTT5_EXAMPLE: key is u, value is user
I (5210) MQTT5_EXAMPLE: key is p, value is password
I (5210) MQTT5_EXAMPLE: payload_format_indicator is 1
I (5220) MQTT5_EXAMPLE: response_topic is /topic/test/response
I (5230) MQTT5_EXAMPLE: correlation_data is 123456
I (5230) MQTT5_EXAMPLE: content_type is
I (5240) MQTT5_EXAMPLE: TOPIC=/topic/qos1
I (5240) MQTT5_EXAMPLE: DATA=data_3
I (5250) mqtt5_client: MQTT_MSG_TYPE_UNSUBACK return code is 0
I (5250) MQTT5_EXAMPLE: MQTT_EVENT_UNSUBSCRIBED, msg_id=61013
I (5260) mqtt_client: Client asked to disconnect
I (6270) MQTT5_EXAMPLE: MQTT_EVENT_DISCONNECTED

I had some issues with the broker not responding to PINGREQ so I have decided to run Wireshark and see what happens.

However, the Wireshark does not capture any MQTT packets for some reason:

I have then downloaded MQTT Explorer tool for windows: http://mqtt-explorer.com/

I have created a MQTT test client device as following:

and as soon as I clicked "Conncet" I got a bunch of messages on Wireshark so that confirms that wireshark is able to capture MQTT packets.

Is there any way to capture MQTT packets from the ESP32 device? Thanks in advance! :)

[ESP-YJM]

@krupis Please confirm that you can use IP address to filter the packet you captured.

[krupis]

@ESP-YJM Yes. I filter out ip address: 192.168.64.106.

That is the IP address that has been assigned to my ESP32 development board ( you can see from the LOGS:

I (4540) example_common: - IPv4 address: 192.168.64.106,

As you can see, no packets are detected.

[ESP-YJM]

@krupis It seems you not capture any packet from ESP32. Could you please share the topology of packet capture.

[krupis]

@ESP-YJM Can you clarify what does topology of packet capture mean? How can I share it with you?

[ESP-YJM]

@krupis That means the PC you use wireshark to capture the packet. In your log, you use ethernet, you should capture the packet with a hub that will mirror the LAN packet to another port you used. Or you can capture the pakcet in router which support capture the packet. Or you also could capture the packet in PC which run MQTT broker. So i wonder the topology you configure to capture.

[krupis]

@ESP-YJM I simply have a router, PC and ESP32 dev board. One cable from the router is connected to my PC, the other cable from the same router connects to the esp32 dev board. Then I simply run wireshark.

Do you think it could be something with the router?

I can try using different network or different router

[ESP-YJM]

@krupis I think it can not capture any packet from ESP32 in PC.Because the ESP32 connected to the LAN is independent with the LAN you PC connected.

[krupis]

@ESP-YJM But I can succesfully ping 192.168.64.106 from my PC and it will respond. Is that not relevant?

Anyways, what is most convenient and reliable way to capture such packets? Do I need to buy some additional equipment? I happen to have NRF52840 board that I used in the past to capture BLE packets. Maybe can use this ?

[krupis]

@ESP-YJM I also decided to try a different method. I use a different router and instead of ethernet I use WiFi.

1. Both my PC and ESP32 are connected to same internet (2.4 Ghz)
2. I run wireshark:
3. On the ESP32 logs, I check which IP address has been assigned to the device:

   
   4291) wifi:connected with TP-Link_5AA5, aid = 1, channel 2, BW20, bssid = b4:b0:24:ec:5a:a5
   I (4291) wifi:security: WPA2-PSK, phy: bgn, rssi: -61
   I (4291) wifi:pm start, type: 1

I (4311) wifi:idx:0 (ifx:0, b4:b0:24:ec:5a:a5), tid:0, ssn:4, winSize:64 I (4321) wifi:AP's beacon interval = 102400 us, DTIM period = 1 2023/05/05 07:44:45: [1B][0;32mI (5301) esp_netif_handlers: example_netif_sta ip: 192.168.0.136, mask: 255.255.255.0, gw: 192.168.0.1[1B][0m [1B][0;32mI (5301) example_connect: Got IPv4 event: Interface "example_netif_sta" address: 192.168.0.136[1B][0m

As you can see from the log above, the ESP32 has been assigned 192.168.0.136
4. I filter this IP address in wireshark and listen for packets:
![image](https://user-images.githubusercontent.com/22237034/236379823-6ebcb89d-e6e8-4601-9ce6-08bbff5d8fcd.png)
Unfortunately, still no packets captured.

[krupis]

I have also read: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/wireshark-user-guide.html

They do not seem to mention using any other equipment other than PC and ESP32 and they somehow manage to capture ESP32 packets.

[ESP-YJM]

If you want to capture wifi air packets, you need a professional packet capture network card. You cannot capture air packets through the wifi network port on the PC side. What you capture is only the incoming and outgoing packets from this network card. And if you want to capture ethernet packet, you need a hub. The MQTT packet is unicast packet, the other lan port will not receive it, so you can't capture it.

For other debug method, you can try this patch on ethrtnet netif and print tcp flow, it still can know the reason why PING_RESP not received. print_tcp_flow.patch

[igrr]

I think it's also possible to put the NIC into monitor mode to capture unicast packets to other devices on the network. In the linked user guide this is achieved using iwconfig wlan0 mode monitor command (Linux). The official wireshark documentation describes how to put a NIC into monitor mode for other OSes: https://wiki.wireshark.org/CaptureSetup/WLAN.

There is also an explanation why capturing Ethernet packets doesn't work with a switch, and alternative solutions: https://wiki.wireshark.org/CaptureSetup/Ethernet#switched-ethernet.

[krupis]

If you want to capture wifi air packets, you need a professional packet capture network card. You cannot capture air packets through the wifi network port on the PC side. What you capture is only the incoming and outgoing packets from this network card. And if you want to capture ethernet packet, you need a hub. The MQTT packet is unicast packet, the other lan port will not receive it, so you can't capture it.

For other debug method, you can try this patch on ethrtnet netif and print tcp flow, it still can know the reason why PING_RESP not received. print_tcp_flow.patch

Please could you clarify what am I supposed to do with that .patch file that you provided? I have never done something like that before.

[krupis]

I think it's also possible to put the NIC into monitor mode to capture unicast packets to other devices on the network. In the linked user guide this is achieved using iwconfig wlan0 mode monitor command (Linux). The official wireshark documentation describes how to put a NIC into monitor mode for other OSes: https://wiki.wireshark.org/CaptureSetup/WLAN.

There is also an explanation why capturing Ethernet packets doesn't work with a switch, and alternative solutions: https://wiki.wireshark.org/CaptureSetup/Ethernet#switched-ethernet.

Il look this up on Monday and respond how it went. Thanks :)

[AxelLin]

Il look this up on Monday and respond how it went. Thanks :)

Any update? @krupis

[krupis]

Il look this up on Monday and respond how it went. Thanks :)

Any update? @krupis

I have tried to put my device into monitor mode by following: https://wiki.wireshark.org/CaptureSetup/WLAN#turning-on-monitor-mode

But without any luck. I think my machine does not have the required hardware to do that.