Very poor latency / FPS performance

[Bartvelp]

I got the ground station running on a laptop, see #39 for more details. But my performance is very bad. I am also no using a diversity receiver, but my latency is atrocious and framerate as well ( 0 to 5 fps). Some logs from the ground station:

(I) src/main.cpp: 102: Sent: 2, RX len: 61740, RSSI: -28, Latency: 2/2/2
(I) src/main.cpp: 102: Sent: 2, RX len: 79380, RSSI: -29, Latency: 4/164/84
(I) src/main.cpp: 102: Sent: 2, RX len: 39690, RSSI: -29, Latency: 177/220/199
(I) src/main.cpp: 102: Sent: 2, RX len: 51450, RSSI: -28, Latency: 180/180/180
(I) src/main.cpp: 102: Sent: 2, RX len: 73500, RSSI: -32, Latency: 39/200/119
(I) src/main.cpp: 102: Sent: 2, RX len: 57330, RSSI: -31, Latency: 103/115/109
(I) src/main.cpp: 102: Sent: 2, RX len: 49980, RSSI: -31, Latency: 76/126/101
(I) src/main.cpp: 102: Sent: 2, RX len: 52920, RSSI: -31, Latency: 134/178/156
(I) src/main.cpp: 102: Sent: 2, RX len: 41160, RSSI: -31, Latency: 145/145/145

and the esp:

WLAN S: 106272, R: 350, E: 0, D: 0, % : 94 || FPS: 51, D: 131983 || D: 0, E: 0
WLAN S: 78228, R: 420, E: 0, D: 0, % : 94 || FPS: 52, D: 131661 || D: 0, E: 0
WLAN S: 76752, R: 420, E: 0, D: 0, % : 97 || FPS: 51, D: 140211 || D: 0, E: 0
WLAN S: 129888, R: 420, E: 0, D: 0, % : 97 || FPS: 52, D: 141822 || D: 0, E: 0
WLAN S: 174168, R: 350, E: 0, D: 0, % : 52 || FPS: 52, D: 141902 || D: 0, E: 0
WLAN S: 64944, R: 420, E: 0, D: 0, % : 97 || FPS: 51, D: 142672 || D: 0, E: 0
WLAN S: 69372, R: 420, E: 0, D: 0, % : 97 || FPS: 52, D: 136611 || D: 0, E: 0
WLAN S: 41328, R: 420, E: 0, D: 0, % : 97 || FPS: 51, D: 132176 || D: 0, E: 0

I am not sure what the issue is, the RSSI seems very good. I am in a noisy 2.4 GHz location, but the proximity should overpower everything. What could be the cause of this performance issue, and how can I diagnose it?

[jeanlemotan]

From what I see, the TX buffer on the esp side is full. Most probably the air data rate cannot keep up with the resolution/FPS and you get a lot of frame drops. In your log, you can see the TX buffer percentage as 94-97%, which is a lot. It should hang at around 0-20% with occasional spikes.

What data rate are you using?

[Bartvelp]

When logging the rate using:

LOG("WLAN S: %d, R: %d, E: %d, D: %d, %%: %d || FPS: %d, D: %d || D: %d, E: %d, WIFI_RATE: %d\n",
  s_stats.wlan_data_sent, s_stats.wlan_data_received, s_stats.wlan_error_count, s_stats.wlan_received_packets_dropped, s_wlan_outgoing_queue.size() * 100 / s_wlan_outgoing_queue.capacity(),
  (int)s_stats.video_frames, s_stats.video_data, s_stats.sd_data, s_stats.sd_drops, get_wifi_fixed_rate());

It gives: WLAN S: 50184, R: 0, E: 0, D: 0, % : 97 || FPS: 25, D: 629476 || D: 0, E: 0, WIFI_RATE: 13, 13 corresponds to RATE_G_54M_ODFM.

This is confirmed by WireShark:

Seems plenty fast to me, weird that the TX buffer is getting filled then.

[jeanlemotan]

In this last log, it's even worse: It's trying to send ~600K of data per second, and it only manages 50K. My guess is that due to the very noisy environment, it cannot find enough free air capacity to send the data. AFAIK the esp still checks for other wifi traffic before sending, even in package injection mode. Try a different channel or modulation. Faster rates might give you better results as they occupy less air time, therefore less chances of collisions. I also did all my tests at home in a super noisy environment and had to choose a more or less free channel to get anything reliable, or drop the FPS/resolution to reduce the air time as much as possible to get good results. Try MCS7 for the data rate, at a lower resolution and watch for buffer overflows. You should be able to tune it for good performance. Note that the firmware doesn't do any sanity checks if you try to fit too much data through the air. There should be some checks added to the ground station when the air rate is too low for the selected resolution/fps combo, but I never got around to adding these.

[Bartvelp]

Your logic makes total sense, but when I run the iperf example, using a softAP I get:
[ 1] 0.0000-60.0055 sec 87.8 MBytes 12.3 Mbits/sec
So that is way more. But when I manually send packets using espnow, I again get max ~80 KB/sec. When running iperf when connected to my regular WiFi I get a bit less performance:
[ 3] 0.0000-59.9971 sec 47.6 MBytes 6.66 Mbits/sec
Still way better than the raw sending. Very weird if you ask me.

[Bartvelp]

I got it working! Turns out I need to touch the backside of the esp32 antenna, and then it starts sending quite well, although still not perfect. You can't really make this stuff up. I'm getting up to 30 FPS but not much beyond, also got really intermittent latency.

[jeanlemotan]

I think that when you touch the backside, you're 'muting' all other noise in the air, so the ESP cannot receive much else, and it thinks it's quiet out there, therefore sending all of its data. You could try to put it in a faraday cage (microwave, aluminum foil, smth) and see if that improves performance.