search

sepfy / pear-esp32-examples

9 stars 2 forks source link

MediaStream and FPS #5

Closed eliabruni closed 1 year ago

eliabruni commented 1 year ago

Hey,

Cool project! I have two questions:

  1. Are you planning to try out the MediaStream API too?
  2. Do you have some performance stats on latency and FPS when transferring images via the data channel?

  3. Thanks!

sepfy commented 1 year ago

Hi, here is my answer

  1. i am developing MediaStream for audio stream. because esp32 does not support hardware video codec, i implemented sending jpeg images through datachannel
  2. In this example, the latency should be 300~400 ms and FPS is 20
eliabruni commented 1 year ago

Hey, thanks for the prompt reply!

  1. Have you seen this: https://components.espressif.com/components/espressif/esp_h264?
  2. Your tests are with FRAMESIZE_VGA which should correspond to 640x480, correct?

Do you think there is any chance I could bring latency down to say 100ms in some way?

sepfy commented 1 year ago
  1. i know there are some ways to do software encode H.264 video on ESP32. But i never test it and check performance. Looks it can encode with 10 fps in low resolution. not sure it can be used in some scenarios? because i think if for user to watch, it should be higher resolution, such as 640x480
  2. Yes

if use chrome as client, i think the limit of latency should be 200~300 ms

eliabruni commented 1 year ago
  1. That's good point.

My use-case is a bit different. Right now I am using aiortc to send image frames from an RPI to an aiortc receiver (no browser). The aiortc receiver then performs some deep learning on the frames. For that to work I would need approx 10 FPS but low latency, with FRAMESIZE_HVGA or FRAMESIZE_VGA. Pear seems to be a great project if I want to try to port my application from RPI to esp32 (btw, I am planning to use this XIAO-ESP32S3-Sense).

Assumed I can reach the desired performance, I would have then to figure out one of the following options I guess:

A. Combining Pear (esp32 side) and aiortc (receiver side). For instance, using the datachannel of your example, I suppose that aiortc could receive binary data over WebRTC B. Porting everything to Pear: that means I would write the receiver side in Pear too, but I am not sure that is possible as it is meant to be used with the Browser?

sepfy commented 1 year ago

i see. with aiortc receiver should be easier, because aiortc already support datachannel. With this esp32 example, when on_message is called in aiortc, a complete jpeg image will be received. Although i am not sure pear can work with aiortc, but datachannel is webrtc standard and it should be easy to fix. if you have connection issue between pear and aiortc, can let me know~ i will try to fix it. Of course, you may need to implement your own signaling

eliabruni commented 1 year ago

Alright, thanks for the feedback and availability for helping out! I will work on this in a couple of weeks and will let you know once I have some results.