search

DJI-Mobile-SDK-Tutorials / Android-VideoStreamDecodingSample

This sample project demonstrates how to use FFmpeg for video frame parsing and to use MediaCodec for hardware decoding on DJI Products.
MIT License
169 stars 80 forks source link

VideoFeeder.VideoDataCallback is never called on dji spark when connected via wifi #43

Closed Consti10 closed 3 years ago

Consti10 commented 5 years ago

Description: It seems like the VideoFeeder.VideoDataCallback is never called when there is no instance of DJICodecManager It is called, but with garbage values if there is an fake instance of DJICodecManager It is only called with data that can be decoded by Android MediaCodec if there is a valid instance of DJICodecManager. Valid, in this context, means configured with a true output surface.

Especially the first one sounds like a bug. If I understand correctly, the purpose of the VideoFeeder.VideoDataCallback is to make .h264 nalus available to the application such that they can be decoded with a custom decoder ( and not the DJICodecManager). This has various use-cases, e.g. my application is tuned for ultra low latency and already has a proven MediaCodec decoder. But I cannot find a way to get rid of the DJICodecManager because of the issues described above. This might also relate to issue #41 , because for the described use-case there is no need to transcode the video into a different bitrate. If thats the purpose of VideoStreamDecodingSample, please clarify that and add the option to register a callback for raw .h264 nalus, as they are coming from the connected drone.

Consti10 commented 5 years ago

I did some more testing: Test 1: VideoFeeder.VideoDataCallback is never called without a DJICodecManager instance only if I connect the app to my sparks rc controller via WIFI When connecting to the controller via usb, it is called - but with non-decodable data Only with a valid DJICodecManager the callback actually provides decodable data

-> Bug 1: non-uniform behavior when using wifi vs usb (at least on the spark- I dont have other models to test with)

Bug 2 When using usb , the VideoFeeder.VideoDataCallback is called, but without decodable data Only using a fake initialized DJICodecManager results in decodable data in the callback

Consti10 commented 5 years ago

Hello,

I have to correct myself, the h264 nalus without initialized DJiCodecManager actually can be decoded, but they are slightly different with an initialized DJICodecManager.

However. the other issue is still existent: No callbacks with spark connected via wifi.

Consti10 commented 5 years ago

For clarification:

I was able to resolve issue part 1 garbage video: It was a bug / unsupported format inside my nalu parser: The dji spark decoder does never send IDR frames, only SPS and PPS nalus. However, the NAL_UNIT_TYPE_CODED_SLICE_NON_IDR== type 1 nalus can be decoded even without key frames - however, you have to make sure to pass them in chuncks of 4 or 6 to the decoder. The chunk size is determined by the AUD's.

But issue part 2 is still existent and probably a bug inside the dji framework: A registered VideoFeeder.VideoDataCallback is called as soon as yoou connect your spark via usb to the controller. But it is never called when connecting via wifi, unless You create a fake DJIDecoder.

This makes it impossible to use a custom decoder with the dji spark (and possible also other dji drones) when connecting via wifi.

Consti10 commented 3 years ago

closing, still not fixed.