search

diux-dev / rosettadrone

Android-based mavlink wrapper for DJI drones
BSD 3-Clause "New" or "Revised" License
113 stars 51 forks source link

Garbled video from DJI drones other than Mavic Pro #3

Closed markdjacobsen closed 5 years ago

markdjacobsen commented 6 years ago

Video feed from P4P and other DJI drones appears garbled in QGroundControl. We thought this may have something to do with I Frames, but it looks like they are being added properly. We have spent extensive time looking for the bug, but have yet to find it.

dbaldwin commented 6 years ago

@jacobsenmd I'm seeing this behavior on my P3 and P4 aircraft. Can you point me to the area of code that you think may be the culprit? I realize you guys have been all over the place trying to figure this out, but if you can give me a starting place I'll dig around this weekend.

markdjacobsen commented 6 years ago

I think it's in the RTP packing. If I send raw H264 from RosettaDrone to gstreamer, and then use gstreamer to pack in RTP and forward to QGC, the video appears fine.

Also, I just remembered there is one known mistake to fix... in DJIVideoStreamDecoder, I am sending H264 frames before I frames are added (by calling frameDataListener.onFrameDataReceived(data, width, height). That call needs to be moved. I fixed it in a local branch but that still didn't resolve the P3/P4 garbled video.

dbaldwin commented 6 years ago

10-4. Thanks for the clarification. Will take a look.

I haven't done much video work on Android so I wanted to ask if there's a way to get the video feed in the emulator. I can do this on iOS but wasn't sure about Android. Or do I need to do it on my actual device and wireless debug with ADB? Any insight would be appreciated.

markdjacobsen commented 6 years ago

This is my effort to summarize troubleshooting thus far.

  1. If we forward raw H264 from Rosetta Drone (rather than RTP-packed) and receive with gstreamer 1.8.3 (running on Ubuntu), the video plays fine:

    gst-launch-1.0 udpsrc port=5600 !  h264parse ! avdec_h264 ! videoconvert ! videorate ! autovideosink

  2. If we instead forward raw H264 from Rosetta Drone and use gstreamer to receive it, RTP-pack it, and re-forward it, QGC plays the video just fine. This led us to think that maybe the bug is in Rosetta Drone's RTP packer.

    
gst-launch-1.0 udpsrc port=5600 !  h264parse  !  rtph264pay ! udpsink host=127.0.0.1 port=5602

3. However, if we forward raw H264 from Rosetta Drone using a Phantom 4 Pro and analyze with ffprobe, we see "decode_slice_header_error" and "non-existing PPS 0 referenced" errors. According to this stackoverflow discussion, this indicates that the decoder hasn't seen a keyframe yet: https://stackoverflow.com/questions/15005436/errors-when-decode-h-264-frames-using-ffmpeg. So perhaps the problem is with the H264 stream (not RTP packing) but gstreamer 1.8.3 is gracefully handling it, while the older version of gstreamer in QGC is not.

By contrast, ffprobe shows no errors when the drone is a Mavic Pro.

[FRAME] media_type=video stream_index=0 key_frame=1 pkt_pts=N/A pkt_pts_time=N/A pkt_dts=N/A pkt_dts_time=N/A best_effort_timestamp=N/A best_effort_timestamp_time=N/A pkt_duration=48000 pkt_duration_time=0.040000 pkt_pos=78679 pkt_size=49989 width=1280 height=720 pix_fmt=yuv420p sample_aspect_ratio=N/A pict_type=I coded_picture_number=23 display_picture_number=0 interlaced_frame=0 top_field_first=0 repeat_pict=0 [/FRAME] [FRAME] media_type=video stream_index=0 key_frame=0 pkt_pts=N/A pkt_pts_time=N/A pkt_dts=N/A pkt_dts_time=N/A best_effort_timestamp=N/A best_effort_timestamp_time=N/A pkt_duration=40000 pkt_duration_time=0.033333 pkt_pos=128668 pkt_size=6225 width=1280 height=720 pix_fmt=yuv420p sample_aspect_ratio=N/A pict_type=P coded_picture_number=24 display_picture_number=0 interlaced_frame=0 top_field_first=0 repeat_pict=0 [/FRAME]



4. We are also exploring the possibility that P4P has a higher bit rate than the Mavic Pro and is overwhelming the computational power of the phone.
sdowhy commented 6 years ago

Fixed in next release.