RTP streaming issues with the DJI Mini SE

[kripper]

Hi @roguestarslade,

Have you tried Rosetta Drone? https://github.com/The1only/rosettadrone

I came to your fork because I'm working on fixing the video stream decoding for the Mini. We are receiving the decoded buffer on onDataRecv() and then send it via RTP. But I believe we are doing something wrong in this process: https://github.com/The1only/rosettadrone/issues/27

[roguestarslade]

Hi @roguestarslade,

Have you tried Rosetta Drone? https://github.com/The1only/rosettadrone

I'm aware of the project, but haven't checked it out recently. Its been a couple years, but I think I can catch up when I have free time.

I came to your fork because I'm working on fixing the video stream decoding for the Mini. We are receiving the decoded buffer on onDataRecv() and then send it via RTP. But I believe we are doing something wrong in this process: The1only/rosettadrone#27

Well well well.... I've been encountering this same error working on the DJI side too. Same stack : a project I'm working on has RTSP, and we see some choppiness coming in from the video on the DJI side. (Im working on this same issue in one of my DJI projects actually. So I'll clue you in...)

Here's what you can do :

• try to find a way to throttle the DJI video stream. I'm not sure how rosettadrone does it, but you need to somehow make DJIs video stream not send so much data.

• DJI side is basically send waaaay too much data. So you'll need to squeeze off that hose, and see what can be fixed from video.

Unfortunately I can't share source due to proprietary stuff I'm working on :/ Best I can do is share clues from the above.

[kripper]

Hi @roguestarslade,

I forked your repo and added code for RTP streaming with a minimal commit: https://github.com/kripper/DJI-Android-VideoStreamDecodingSample/commit/9d31891398c412b589f5820fc52cebc519c2ac5c

This code is supposed to be working fine for other models, but not for the Mini.

[kripper]

What I found strange is that the sample code obtains a keyframe using this magical getIframeRawId() function and this keyframe is added only once!

hasIFrameInQueue is set to false only in releaseCodec() and stop().

[kripper]

I updated the code to enable streaming RTP with Unicast (Multicast didn't work for us). https://github.com/roguestarslade/DJI-Android-VideoStreamDecodingSample/commit/6924305079400747a7a16626fe717e6caf486b5a

In our code, we split the NALs to solve the problem of sending to big UDP packets, which are automatically splitted and received without headers.

In this commit, I added a boolean "splitNALs" that can be modified using the debugger to disable the NAL splitting during runtime. I discovered that disabling this boolean after the client started to receive the stream, improved the quality significantly, which means that we are probably doing something wrong in the NAL splitting code. Without NAL splitting, gstreamer was not able to open the stream at all.

[roguestarslade]

I updated the code to enable streaming RTP with Unicast (Multicast didn't work for us). 6924305

In our code, we split the NALs to solve the problem of sending to big UDP packets, which are automatically splitted and received without headers.

In this commit, I added a boolean "splitNALs" that can be modified using the debugger to disable the NAL splitting during runtime. I discovered that disabling this boolean after the client started to receive the stream, improved the quality significantly, which means that we are probably doing something wrong in the NAL splitting code. Without NAL splitting, gstreamer was not able to open the stream at all.

hey @belveder79 check this out.

[belveder79]

hmmm... @roguestarslade my assumption is that at this point

https://github.com/roguestarslade/DJI-Android-VideoStreamDecodingSample/blob/fe6fa101945e3ffbe94d7c577d25e7d38bb1962d/android-videostreamdecodingsample/app/src/main/java/com/dji/videostreamdecodingsample/media/DJIVideoStreamDecoder.java#L168

the stream from the Drone was received correctly, which means that the buffer contains valid NAL units (and only complete NAL units). What we do next is hand that buffer over to a streaming library, and this library creates a new RTP stream out of that and wraps it into RTSP, therefore the underlying library is responsible for not overshooting the packet size that can be reliably used over TCP or UDP. There is no custom code required to split NAL units... if the splitting in the streaming library itself wouldn't work (respectively the assembly on the receiving end), you would never get a frame at all...

Now what is an obvious issue is that if TCP is not used, the packets might get lost or the order is not preserved, which is a network issue and you can't really do anything about it. Assuming that your receiving device is actually close to the stream destination or you have a good quality of connection between the receiving device (the DJI app) and your final end point, having issues like that are unlikely to occur imho.

What is less obvious (and I did not dive into that too much to be honest) is that my assumption about the status of the stream at the above mentioned position is wrong. Say there are valid NAL units, but there are also incomplete NAL units, as - for whatever reason - it is just a snapshot of the underlying receive buffer from DJI. Whatever is after the last valid NAL unit is discarded then and you would loose some data (namely exactly the unit that falls on the border between two calls to handover message to the streaming library). This is actually a pretty unusual behaviour that would never occur if you get the stream straight from an encoder, but who knows?

https://github.com/roguestarslade/DJI-Android-VideoStreamDecodingSample/blob/fe6fa101945e3ffbe94d7c577d25e7d38bb1962d/android-videostreamdecodingsample/jni/dji_video_jni.c#L181

I will have a look at this... probably this is what happens at least on our side...

[kripper]

Yes, the frames are correctly enqueued, dequeued, decoded and presented on the screen. Even when you discard 59 frames from 60, you will still get a perfect video on the screen. The NALU splitting seems to be necessary for RTP streaming, because otherwise the receiver (gstreamer) complains of an invalid stream.

Have you tried my fork? What aircraft model are you using? https://github.com/kripper/DJI-Android-VideoStreamDecodingSample

[belveder79]

No I did not as I was just on the project about the streaming... what do you mean by discarding 59 out of 60 frames? Are you talking about NAL units?

[kripper]

We did some tests just ignoring the callback that receives the buffer from the SDK (eg. just processing the data once each 60 times), and the video was still showing almost perfect on the screen.

My understanding is that this buffer contains multiple NAL units and that they must be split before sending them via UDP.

[belveder79]

sorry, I think that there is a misunderstanding or we talk past each other, because what you are saying does not make any sense to me...

of course you need to split NAL units, even individual ones, such that they can be put into RTP packets meeting the requirements for reliable network transfer... however, you cannot just skip calls to

https://github.com/roguestarslade/DJI-Android-VideoStreamDecodingSample/blob/fe6fa101945e3ffbe94d7c577d25e7d38bb1962d/android-videostreamdecodingsample/jni/dji_video_jni.c#L114

and claim to get a perfect video, otherwise the data would have to be redundant, which it certainly is not...

[kripper]

We are parsing in the same way as the example. We split and send the NALUs on the message dequeue code just before the buffer is sent to the MediaCodec decoder for the onscreen rendering.

I'm not saying that the data is redundant, but confirming that the video buffer if received correctly, that it is tolerant and that whatever problem we may experience on the RTP receiver is not caused by missing or dropped packets but by some kind of corruption we are introducing probably in the NALU splitting process which may be incompatible with the specific format of the Mini SE (other models work fine).

Please note I don't know how you are RTP streaming and haven't seen your code. I'm talking about the RTP streaming code I published here: https://github.com/kripper/DJI-Android-VideoStreamDecodingSample

[belveder79]

Ah, ok, now I got it...

Quickly browsing through your code I realized that you never call setStreamParameters on the Packetizer. You could try to filter the first NAL unit with type 7 and 8 and pass that as SPS and PPS bytes before sending the first packet, probably the Packetizer needs some information ahead of time...

what does the SetUseDualVideoOut do? It appears to me it sends the data twice over two ports? Is that required?

[kripper]

SetUseDualVideoOut is used by Rosetta to send the video to two receivers using Unicast. It is not required.

Are you receiving the stream correctly? What model are you testing?

[roguestarslade]

Are you receiving the stream correctly? What model are you testing?

We can receive the stream from a Mavic 2 Pro and Mini 2

[kripper]

Thanks. Can you please confirm you are using my fork (https://github.com/kripper/DJI-Android-VideoStreamDecodingSample) and which gstreamer command line you used to receive the stream?

[kripper]

@roguestarslade can you please confirm if you are referring to my fork and which gstreamer command line you used for receiving the stream? Thanks.

[kripper]

@roguestarslade Just in case, I added detailed instructions how to test the RTP Streaming: https://github.com/kripper/DJI-Android-VideoStreamDecodingSample/blob/develop/README.md