search

fpv-wtf / dji-moonlight-embedded

GNU General Public License v3.0
10 stars 2 forks source link

Feature Request: Stream CSI input to shim #3

Closed CorruptName closed 1 year ago

CorruptName commented 1 year ago

Not sure if this is possible as is but I'm curious if we can do something like this:

Use HDMI -> MIPI Adapter for raspberry pi (Amazon Link) Stream HDMI input to shim This would give actual HDMI input - though with a bit more latency. We could then use Raspberry Pi Zero 2 or similar to make a small box to input HDMI over USB-C

Awesome Work Either Way!

j005u commented 1 year ago

We need to feed the Goggles h264 video, not a raw display stream. Meaning the PI would have to encode 810p@120fps, which I doubt is going to happen. And with any such solution you're going to have quite a bit of added latency.

The best bet is to work with an HDMI capture card that can output h264 directly and simply pipe that through a PI into the Goggles. With a little bit of extra grunt it'd also be possible to make the Goggles talk directly to a capture card via USB.

Latency would still be a concern in the 2nd case as well.

CorruptName commented 1 year ago

Ah okay. I Still think the best solution to get actual HDMI input to the DJI Goggles would be to connect a HDMI MIPI adapter to the VTX in place of the camera. Trouble is actually implementing this is more than I have time/skill to do.

Now that I say that, would it be possible to pipe the Moonlight data into the VTX USB-C port and then over-air to the goggles? That way you are untethered... ? Is the VTX even true USB 3.0? Never paid much attention to transfer speed...

We need to feed the Goggles h264 video, not a raw display stream. Meaning the PI would have to encode 810p@120fps, which I doubt is going to happen. And with any such solution you're going to have quite a bit of added latency.

The best bet is to work with an HDMI capture card that can output h264 directly and simply pipe that through a PI into the Goggles. With a little bit of extra grunt it'd also be possible to make the Goggles talk directly to a capture card via USB.

Latency would still be a concern in the 2nd case as well.

EDIT: It would also be cool if we could pass network data through an air unit to the goggles. Big Dreams...

Edit 2: Which USB Capture card would you suggest for option 2? And if you would like to try this lmk if you need a capture card.

j005u commented 1 year ago

Ah okay. I Still think the best solution to get actual HDMI input to the DJI Goggles would be to connect a HDMI MIPI adapter to the VTX in place of the camera. Trouble is actually implementing this is more than I have time/skill to do.

Yeah the somewhat tricky bit is reverse engineer the exact required MIPI stream config from the RTOS and the making sure your camera/HDMI converter can match it. But it should be doable.

Now that I say that, would it be possible to pipe the Moonlight data into the VTX USB-C port and then over-air to the goggles? That way you are untethered... ? Is the VTX even true USB 3.0? Never paid much attention to transfer speed...

It's USB2, but so are the Goggles. 480mbit is more than enough and then some for 810p120 h264.

We need to feed the Goggles h264 video, not a raw display stream. Meaning the PI would have to encode 810p@120fps, which I doubt is going to happen. And with any such solution you're going to have quite a bit of added latency. The best bet is to work with an HDMI capture card that can output h264 directly and simply pipe that through a PI into the Goggles. With a little bit of extra grunt it'd also be possible to make the Goggles talk directly to a capture card via USB. Latency would still be a concern in the 2nd case as well.

EDIT: It would also be cool if we could pass network data through an air unit to the goggles. Big Dreams...

We can! The Goggles are 192.168.41.1 and air side is 192.168.41.2. TCP and UDP are both supported. TCP can be a bit wonky. This is how msp-osd works and it's also enough to stream live audio via ffmpeg to the v2 goggles. Also the goggles have adb on them and you can adb connect between rooted devices over IP.

It's still pretty slow though, at least while the video stream is running. I don't have exact numbers, since it also depends if you're using TCP or UDP and we've had varying results. Also I don't think anyone's tested what kind of BW you can push if you disconnect the camera, therefor disabling the video stream.

If you add http://bin.entware.net/armv7sf-k3.2/ to your opkg sources in /opt/etc/opkg/ then you should be able to install iperf. Note the http, the configurator proxy will do a translation to https. I'd remove the source after, can't say for sure you won't get issues with our partial entware repo mirror otherwise later. Or you know, build from source with the NDK clang.

Edit 2: Which USB Capture card would you suggest for option 2? And if you would like to try this lmk if you need a capture card.

I saw someone mention Elgato stuff in passing in relation to this topic, but I couldn't make any specific recommendations. It should be able to handle at least 810p120, output actual h264 not just mjpeg and ideally have the lowest possible latency.

CorruptName commented 1 year ago

Very cool stuff. I will definitely have to make time to mess around with it. I think it would be very cool if make a dedicated box with an air unit and we could pass h264 into the VTX (Jetson would be very good for encoding!) then output OTA to the goggles. The applications for this are numerous - especially if we could find a capture unit that would output h264 like you said. add a stable internet connection and then things get really interesting. Well done folks.