search

DroneBridge / ESP32

DroneBridge for ESP32. A secure & transparent telemetry link with support for WiFi and ESP-NOW. Supporting MAVLink, MSP, LTM or any other protocol
https://dronebridge.github.io/ESP32/
Apache License 2.0
412 stars 113 forks source link

Support RTSP with JPEG-capable cameras #73

Open mmalecki opened 6 months ago

mmalecki commented 6 months ago

I know the README mentions not being to support cameras due to lack of ESP32's video encoding capabilities. However, it appears that RTSP with JPEG-capable (and thus MJPEG-capable) cameras is something ESP32 is able to do:

QGroundControl is also able to read in RTSP streams, leading me to assume that ESP32 streaming video to QGroundControl is a real possibility.

Since I already started working on integrating this, I suppose the question is would you be open to a PR if I succeed (hidden behind the appropriate #ifs, of course)?

Edited to add: it now occurs to me that QGC may not be able to read and display MJPEG, so this may need additional GStreamer pipeline running on receiving side. Will double-check this in course of implementation.

Edited to add: I just noticed your comments in #8 about this set-up not offering any improvements over analog. In my use-case, I also want to do some image processing on the ESP32 itself, so analog does not work for me, but feel free to close this issue.

seeul8er commented 6 months ago

I still see integration of ESP32 cameras into DroneBridge for ESP32 critical. Where is the benefit compared to the already existing implementations? What would be the use-case? As I see it, the framerate is very low and latency not so great.

I will not invest any resources into porting or supporting ESP32 cam support for now. What might be worth the effort is the upcoming ESP32-P4 chip. It will support H.264 HW encoding and seems to be a better fit overall. Let's see what kind of API they will provide and how it performs.

mmalecki commented 6 months ago

Where is the benefit compared to the already existing implementations?

What would be the use-case?

For me, the use-case is simple. I don't fly FPV, but I enjoy leisure-y cruising and hacking on my drone, and I want to experiment with a bit of machine learning with it. I picked ESP32-S3 due to my previous experience with it, and its ability to run basic ML models. And so, the drone already has a ESP32-S3 with a camera on it. I picked DroneBridge due to its flight record and nice architecture. And so, connecting these parts only made sense to me. A more generalized use-case will be pretty similar, I imagine: "have ESP-CAM, run DroneBridge already, want the camera footage transmitted as well". I'm also looking to release a PCB that has both the camera and a PX4 standard FC connector, primarily to make my own life easier, and being able to point at a ready-made piece of software for users to flash onto it would make their lives easier, too.

As I see it, the framerate is very low and latency not so great.

Due to my use-case, this is acceptable for me. Not sure if it will be for the rest of the user-base. Real win here, IMO, would be to implement a raw WiFi protocol (to the tune of aforementioned esp32-cam-fpv, or the wfb-ng protocol, which I seem to recall you've considered before). ACK frames really are a menace for this use-case.

I will not invest any resources into porting or supporting ESP32 cam support for now.

That makes sense. Would you be open to merging a PR for adding optional support?

What might be worth the effort is the upcoming ESP32-P4 chip. It will support H.264 HW encoding and seems to be a better fit overall. Let's see what kind of API they will provide and how it performs.

Super excited for that. Dual camera support is just :cherries: :cake:.

seeul8er commented 6 months ago

I understand, however the use-case is still rather focused on tinkerers and people experimenting. I am a bit afraid that the additional code might add a lot of complexity which only a few users will actually use. My goal for this project was to have an easy to use, clean and performant system.

How about creating a separate branch that features support for such camera modules? Lets see if I can also give you direct access to it. That way it is stilled tied to the base project. It can be mentioned in the README and Wiki. Also the releases can be published alongside the regular DroneBridge for ESP32 releases. That way the original code base will continue to cover the general use-case of most users, while the branch can focus on the more specialized scenarios.

Once the ESP32-P4 chip with integrated camera modules becomes available I would integrate video transmission as a default feature into DroneBridge for ESP32. Also alongside FEC and RAW-Wifi transmission like with the DroneBridge for Raspberry Pi project. Since it will be a proper embedded system with only few external dependencies I'd imagine it may even be superior to the Raspberry Pi based solutions.

mmalecki commented 5 months ago

I understand, however the use-case is still rather focused on tinkerers and people experimenting. I am a bit afraid that the additional code might add a lot of complexity which only a few users will actually use. My goal for this project was to have an easy to use, clean and performant system.

Totally understood! That's what brought me to this project in the first place :)

How about creating a separate branch that features support for such camera modules? Lets see if I can also give you direct access to it. That way it is stilled tied to the base project. It can be mentioned in the README and Wiki. Also the releases can be published alongside the regular DroneBridge for ESP32 releases.

Thank you, appreciate this for sure! I'll continue development in my branch and let you know how it goes.

Super happy to hear you plan on supporting ESP32-P4!