search

jeanlemotan / esp32-cam-fpv

esp32 cam digital low latency fpv
MIT License
278 stars 68 forks source link

I end up with an end-to-end latency of around 800 milliseconds #12

Open vguenther opened 1 year ago

vguenther commented 1 year ago

I tried to get this running, but found several issues, which might or might not be related to each other. Any idea what's going wrong?

Issues:

  1. I end up with an end-to-end latency of around 800 milliseconds – far beyond the figures in the description. I found this out by by directing the camera to the display and at the same time to a stopwatch app and taking a photo of the setup.

  1. The output on the display shows checkerboard-like compression artefacts and is virtually unusable.
  2. Though I can move the cursor on the RPI-screen with the mouse, I cannot change the configuration-settings.
  3. gs shows error messages on startup (I added LOGE("{}", SDL_GetError()); in the SDLCHK macro)

pi@fpvbusterpi:~/esp32-cam-fpv/gs $ sudo -E LD_LIBRARY_PATH=/usr/local/lib ./gs (I) src/PI_HAL.cpp: 125: Initializing pigpio (I) src/PI_HAL.cpp: 283: Drivers: 4 (I) src/PI_HAL.cpp: 286: Driver 0: x11 (I) src/PI_HAL.cpp: 286: Driver 1: KMSDRM (I) src/PI_HAL.cpp: 286: Driver 2: dummy (I) src/PI_HAL.cpp: 286: Driver 3: evdev (I) src/PI_HAL.cpp: 295: Mode 0: 1280x1024 (I) src/Comms.cpp: 718: Radiocap header size: 11, IEEE header size: 24 (I) src/Comms.cpp: 581: Opening interface wlan1 in monitor mode (I) src/Comms.cpp: 282: DLT_IEEE802_11_RADIO Encap (I) src/Comms.cpp: 581: Opening interface wlan2 in monitor mode (I) src/Comms.cpp: 282: DLT_IEEE802_11_RADIO Encap (I) src/Video_Decoder.cpp: 160: SDL window: 31254080 (E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161 (E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS) (I) src/Video_Decoder.cpp: 160: SDL window: 31254080 (E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161 (E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS) (I) src/Video_Decoder.cpp: 160: SDL window: 31254080 (E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161 (E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS) (I) src/Video_Decoder.cpp: 160: SDL window: 31254080 (E) src/Video_Decoder.cpp: 161: SDL error -1 in SDL_GL_MakeCurrent(m_impl->window, m_impl->contexts[thread_index])file src/Video_Decoder.cpp line 161 (E) src/Video_Decoder.cpp: 161: Unable to make EGL context current (call to eglMakeCurrent failed, reporting an error of EGL_BAD_ACCESS)

What's my setup and what did I do?

  • I’m using an RPI4 with a Buster image and TL-WN722 v2 adapters together with appropriate driver from https://github.com/aircrack-ng/rtl8188eus. • I provided the prerequisites as described • I fixed the obvious issue in Makefile in order to find all headers • I had to change Comms.cpp ( pcap_set_rfmon(pcap.pcap, 0) < 0) ) in order to work around a PCAP problem (pcap_activate … too many arguments) • WLAN1 und WLAN2 are successfully brought up in monitoring mode before startup of gs • I made sure ESP32-CAM was running continuously • I accessed RPI by ssh to start gs • I did serial monitoring on ESP32

Typical output of gs:

… (I) src/main.cpp: 104: RX len: 77910, RSSI: -65, Latency: 158/158/158 (I) src/main.cpp: 104: RX len: 99960, RSSI: -69, Latency: 144/144/144 (I) src/main.cpp: 104: RX len: 104370, RSSI: -47, Latency: 168/168/168 (I) src/main.cpp: 104: RX len: 113190, RSSI: -81, Latency: 162/162/162 (I) src/main.cpp: 104: RX len: 110250, RSSI: -47, Latency: 130/130/130 (I) src/main.cpp: 104: RX len: 108780, RSSI: -65, Latency: 140/140/140 (I) src/main.cpp: 104: RX len: 104370, RSSI: -69, Latency: 144/144/144 (I) src/main.cpp: 104: RX len: 107310, RSSI: -47, Latency: 167/167/167 (I) src/main.cpp: 104: RX len: 98490, RSSI: -47, Latency: 160/160/160 (I) src/main.cpp: 104: RX len: 108780, RSSI: -81, Latency: 195/195/195 (I) src/main.cpp: 104: RX len: 97020, RSSI: -65, Latency: 126/126/126 (I) src/main.cpp: 104: RX len: 102900, RSSI: -95, Latency: 180/180/180 (I) src/main.cpp: 104: RX len: 101430, RSSI: -95, Latency: 130/130/130 (I) src/main.cpp: 104: RX len: 104370, RSSI: -95, Latency: 141/141/141 …

HAL widget on DISPLAY reports <5ms/frame and >200fps

Typical output of esp32-cam:

… MEMORY Before Loop: Heap summary for capabilities 0x00000004: At 0x3f800000 len 4194303 free 979859 allocated 3211264 min_free 979859 largest_free_block 966656 alloc_blocks 257 free_blocks 1 total_blocks 258 At 0x3ffae6e0 len 6432 free 4 allocated 4136 min_free 4 largest_free_block 0 alloc_blocks 35 free_blocks 0 total_blocks 35 At 0x3ffb7920 len 165600 free 4 allocated 162832 min_free 4 largest_free_block 0 alloc_blocks 153 free_blocks 0 total_blocks 153 At 0x3ffe0440 len 15072 free 4 allocated 12876 min_free 4 largest_free_block 0 alloc_blocks 10 free_blocks 0 total_blocks 10 At 0x3ffe4350 len 113840 free 6888 allocated 104528 min_free 6872 largest_free_block 6784 alloc_blocks 68 free_blocks 2 total_blocks 70 Totals: free 986759 allocated 3495636 min_free 986743 largest_free_block 966656 WLAN S: 29520, R: 70, E: 0, D: 0, % : 99 || FPS: 1, D: 68846 || D: 0, E: 0 Wifi rate changed from 9 to 13 Camera resolution changed from 3 to 0 Target FPS changed from 30 to 0 Camera quality from 8 to 63 WLAN S: 119556, R: 280, E: 0, D: 0, % : 97 || FPS: 48, D: 537492 || D: 0, E: 0 WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 54, D: 123820 || D: 0, E: 0 WLAN S: 119556, R: 350, E: 0, D: 0, % : 95 || FPS: 54, D: 124757 || D: 0, E: 0 WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 54, D: 124719 || D: 0, E: 0 WLAN S: 121032, R: 210, E: 0, D: 0, % : 95 || FPS: 54, D: 125311 || D: 0, E: 0 WLAN S: 119556, R: 350, E: 0, D: 0, % : 97 || FPS: 53, D: 125379 || D: 0, E: 0 WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 54, D: 125704 || D: 0, E: 0 WLAN S: 119556, R: 210, E: 0, D: 0, % : 97 || FPS: 53, D: 123120 || D: 0, E: 0 WLAN S: 121032, R: 280, E: 0, D: 0, % : 95 || FPS: 53, D: 122297 || D: 0, E: 0 WLAN S: 119556, R: 210, E: 0, D: 0, % : 95 || FPS: 53, D: 124083 || D: 0, E: 0 WLAN S: 119556, R: 280, E: 0, D: 0, % : 95 || FPS: 53, D: 125109 || D: 0, E: 0 WLAN S: 119556, R: 280, E: 0, D: 0, % : 95 || FPS: 53, D: 124289 || D: 0, E: 0 WLAN S: 119556, R: 350, E: 0, D: 0, % : 97 || FPS: 53, D: 124572 || D: 0, E: 0 WLAN S: 121032, R: 350, E: 0, D: 0, % : 95 || FPS: 53, D: 123323 || D: 0, E: 0

…

jeanlemotan commented 1 year ago

Hi I don't know what could be going on honestly. It looks like something on the PI side - it's buffering a lot of frames for some reason. The GL errors could be relevant for this. I cannot debug this on my side as I don't have time ATM, but I'll check with the first opportunity.

Regarding the mouse, it's probably an issue on the ImGUI -> SDL interaction, where it doesn't register the mouse as a valid input device. Probably trivial to fix if you can debug the issue.

jeanlemotan commented 1 year ago

What Pi are you using for the GS?

vguenther commented 1 year ago

Rpi4 4GB with Buster

JulesThuillier commented 1 year ago

Hello ! I am getting the same kind of error (Video_Decoder line 161), black appears with a few control sliders in a corner. I see a frame rate updating for a few seconds then I have a sigHandler message in the terminal. Have you managed to fix it ?

vguenther commented 1 year ago

…. Have you managed to fix it ?

No, I haven‘t.

jeanlemotan commented 1 year ago

I managed to compile and setup the pi to debug this issue, I will check tomorrow in detail.