Unable to decode m300 video with ffmpeg

[idani301]

Hi, we are trying to decode the m300 videos received by the method VideoFeeder.getInstance().getPrimaryVideoFeed.addVideoDataListener() VideoFeeder.getInstance().getSecondary.getSecondaryVideoFeed().addVideoDataListener();

By using ffmpeg.

even the sample https://github.com/DJI-Mobile-SDK-Tutorials/Android-VideoStreamDecodingSample is unable to decode those bytes.

For the main camera we currently use provideTranscodedVideoFeed() so it's alright, but we can't decode remotely the FPV or the secondary camera (we use dual gimbal for m300).

Is there any way to decode those packets? Maybe the packets are in h265 protocol and we should use different method for decoding?

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #42771:

Dear Customer,

Thank you for contacting DJI. For the M300RTK, need to assign(assignSourceToPrimaryChannel) the video channel, here is a sample, and please refer to it. https://github.com/dji-sdk/Mobile-SDK-Android/blob/master/Sample%20Code/app/src/main/java/com/dji/sdk/sample/demo/camera/VideoFeederView.java

Thanks,

Luce Luo DJI Developer Support

[idani301]

Thank you very much for the response Luce Luo.

We were using assignSourceToPrimaryChannel already.

Our problem is that we receive bytes from the following functions , but we cannot decode them. Not by using FFmpeg or even https://github.com/DJI-Mobile-SDK-Tutorials/Android-VideoStreamDecodingSample

VideoFeeder.getInstance().getPrimaryVideoFeed.addVideoDataListener() VideoFeeder.getInstance().getSecondary.getSecondaryVideoFeed().addVideoDataListener()

@Override public void onReceive(byte[] bytes, int i) { // We receive here bytes, how we decode them? The older ways we used on previous drones not working with m300. }

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #42771:

Dear Customer,

Thank you for contacting DJI. Oh, sorry. In general, the FPV is not open, and can't access the data.

Can you tell me what cameras mounted? Have you succeeded to receive the data buff from both primary and second view? Or is there no data in the callback though mounted the second camera?

And the byte in the 'onReceive' callback is H264 format. You can use FFmpeg or other tools to decode it theoretically.

Thanks,

Luce Luo DJI Developer Support

[idani301]

We tried the following configurations :

1. M300 with XT2 on the primary(Left) and Z30 on the secondary.
2. M300 with XT2 on the primary without Z30
3. M300 with Z30 on the primary

We receive bytes on the primary video feed and secondary video feed VideoFeeder.getInstance().getPrimaryVideoFeed.addVideoDataListener() VideoFeeder.getInstance().getSecondary.getSecondaryVideoFeed().addVideoDataListener();

When using the first configuration the data rates we measure are (In MB/s):

2021-01-11 14:04:48.701 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Primary : 3.997848 2021-01-11 14:04:48.701 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Secondary : 6.544336 2021-01-11 14:04:49.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Primary : 3.656184 2021-01-11 14:04:49.701 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Secondary : 6.305536 2021-01-11 14:04:50.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Primary : 3.594448 2021-01-11 14:04:50.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Secondary : 6.189272

When we repeat the exact same experiment on our M210 with XT2 on left and Z30 on right we receive those data rates in MB/s

2021-01-11 14:02:57.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Primary : 3.71348 2021-01-11 14:02:57.701 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Secondary : 1.861824 2021-01-11 14:02:58.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Primary : 3.191616 2021-01-11 14:02:58.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Secondary : 1.731176 2021-01-11 14:02:59.700 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Primary : 2.796832 2021-01-11 14:02:59.701 7721-7807/io.eyesatop.dji.apps I/System.out: Total Video for Secondary : 1.80412

For some reason we easily decode the bytes from the M210 using XT2 and Z30, but on M300 we cannot.

You can easily see the difference when connecting https://github.com/DJI-Mobile-SDK-Tutorials/Android-VideoStreamDecodingSample To M300 and then M210.

Press the button "Demo custom decoder" on your sample and see that on M300 you cannot see video, and on M210 with the same camera you can.

The "Demo custom decoder on the dji sample sends those bytes to FFMPEG by using NativeHelper.getInstance().parse(buf, msg.arg1); and receive result on void onDataRecv(byte[] data, int size, int frameNum, boolean isKeyFrame, int width, int height)

On M300 the onDataRecv returns width and height 0. on M210 the onDataRecv returns good frames.

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #42771:

Dear Customer,

Thank you for contacting DJI. Ok, it still needs to investigate further. Please give us an email via the form, will handle it in another thread. https://formcrafts.com/a/dji-developer-feedback-en

Thanks,

Luce Luo DJI Developer Support

[IgMart]

Hi @idani301 ,

Have you been able to decode the video from the XT2 in M210? I am just able to decode one video (thermal or optical) but I cannot receive both of them at the same time. I have it in the primary gimbal.

Thanks

[idani301]

Yea for some reason you cannot receive both thermal and optical from XT2 at the same time even though DJI has bandwidth to do so. The setDisplayMode on camera number 2 is the only way I found to just toggle the display of the XT2 or Mavic 2 Dual.