VideoFrameProcessor delegate does not work consistently when using Mavic Mini

[AaaK00]

Our app is using DJI SDK and DJI Widget for getting individual frames from the Video Steam. I recently updated our DJI SDK to 4.13 & DJIWidget to 1.6.3 and started testing the App with Mavic Mini. It seems that HW decoding does not work in a similar manner on Mavic Mini than for example on Mavic 2 Enterprise.

After Aircraft connects to App, our code does following actions (Simplified version):

DJIVideoPreviewer.instance().registFrameProcessor(self)
DJIVideoPreviewer.instance().type = .none
DJIVideoPreviewer.instance().enableHardwareDecode = true
DJIVideoPreviewer.instance().enableFastUpload = true
DJIVideoPreviewer.instance().encoderType = ._MavicMini
DJIVideoPreviewer.instance().start()
DJISDKManager.videoFeeder()?.primaryVideoFeed.add(self, with: nil)

Our DJIVideoFeedListener Implementation:

extension XYZ: DJIVideoFeedListener {
    func videoFeed(_ videoFeed: DJIVideoFeed, didUpdateVideoData videoData: Data) {
        let videoData = videoData as NSData
        let videoBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: videoData.length)
        videoData.getBytes(videoBuffer, length: videoData.length)
        DJIVideoPreviewer.instance().push(videoBuffer, length: Int32(videoData.length))
    }
}

And our VideoFrameProcessor Implementation:

extension XYZ: VideoFrameProcessor {
    func videoProcessorEnabled() -> Bool {
        return true
    }
    func videoProcessFrame(_ frame: UnsafeMutablePointer<VideoFrameYUV>!) {
        if frame.pointee.cv_pixelbuffer_fastupload != nil {
            // Create UIImage
        }
    }
}

The problem I'm encoutering with Mavic Mini is that the videoProcessFrame is not called with similar parameters always. frame.pointee.cv_pixelbuffer_fastupload is other than nil only right after booting the aircraft. If I for example kill my app and relaunch it, the videoProcessFrame gets called with each frame, but frame.pointee.cv_pixelbuffer_fastupload is nil and it seems that HW decoding is not working.

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #37910:

Dear Customer,

Thank you for contacting DJI. I checked your code, please try to set the HW to true and test it again.

DJIVideoPreviewer.instance().enableHardwareDecode = true

Thanks,

Luce Luo DJI Developer Support

[AaaK00]

Agent comment from Luce Luo in Zendesk ticket #37910:

Dear Customer,

Thank you for contacting DJI. I checked your code, please try to set the HW to true and test it again.

DJIVideoPreviewer.instance().enableHardwareDecode = true

Thanks,

Luce Luo DJI Developer Support

Ah sorry, I had copy - pasted incorrect snippet to the issue. Now it's fixed in the original post.

As you can see we are already using enableHardwareDecode = true but it does not help.

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #37910:

Dear Customer,

Thank you for contacting DJI. Do you want to implement your own process logic? You can locate the source file in the DJIWidget and modify it by yourself. ​

Thanks,

Luce Luo DJI Developer Support
inline167230393.png

[AaaK00]

Nope, that's not what I want.

I'd like to be able to use the same logic for processing frames in DJI Mavic Mini as I'm using with other aircrafts. As said, our current App works as expected at least with Mavic 2 Enterprise, Mavic 2 Zoom and Matrice 210. Haven't tested other models.

Does your response imply that the issue I'm facing is known, and there's no solution for it atm?

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #37910:

Dear Customer,

Thank you for contacting DJI. I see and I will escalate this issue to the engineer to investigate further. I will cost some time and we will contact you again once any progress.

Thanks,

Luce Luo DJI Developer Support

[zyphs21]

@AaaK00 Hi. There is a lot of discussion in DJIWidget issues 9 about the frame.pointee.cv_pixelbuffer_fastupload is nil, even though enable HardwareDecode.

So it could be a better way to deal with the nil situation.

Here is my code:

func videoProcessFrame(_ frame: UnsafeMutablePointer<VideoFrameYUV>!) {
        let resolution = CGSize(width: CGFloat(frame.pointee.width), height: CGFloat(frame.pointee.height))

        if frame.pointee.cv_pixelbuffer_fastupload != nil {
            //  cv_pixelbuffer_fastupload to CVPixelBuffer
            let cvBuf = unsafeBitCast(frame.pointee.cv_pixelbuffer_fastupload, to: CVPixelBuffer.self)

        } else {
            // create CVPixelBuffer by your own, createPixelBuffer() is an extension function for VideoFrameYUV
            let pixelBuffer = frame.pointee.createPixelBuffer()
            guard let cvBuf = pixelBuffer else { return }
        }
 }

as above, the createPixelBuffer() is an extension function for VideoFrameYUV, you can check the code at VideoFrameYUV+Extension.swift which is in my project

[AaaK00]

func videoProcessFrame(_ frame: UnsafeMutablePointer<VideoFrameYUV>!) {
        let resolution = CGSize(width: CGFloat(frame.pointee.width), height: CGFloat(frame.pointee.height))

        if frame.pointee.cv_pixelbuffer_fastupload != nil {
            //  cv_pixelbuffer_fastupload to CVPixelBuffer
            let cvBuf = unsafeBitCast(frame.pointee.cv_pixelbuffer_fastupload, to: CVPixelBuffer.self)

        } else {
            // create CVPixelBuffer by your own, createPixelBuffer() is an extension function for VideoFrameYUV
            let pixelBuffer = frame.pointee.createPixelBuffer()
            guard let cvBuf = pixelBuffer else { return }
        }
 }

Thanks for the hint @zyphs21!

Yeap, I'm aware of that issue. Strange thing is that I'm not seeing it at all on other devices than Mavic Mini. HW Decoding works as expected on Mavic 2 Enterprise, Mavic 2 Zoom and Matrice 210.

Unfortunately that workaround does not solve the issue for us since the other features that exist on our app are pretty CPU intensive. That effectively means that we need to be able to use HW decoding, otherwise app does not have enough resources for its execution.

[sgallo0692]

@AaaK00 did you ever get this working?

[AaaK00]

@AaaK00 did you ever get this working?

Didn't get working in a sense that the decoding would be done using GPU. Software based decoding works with the workaround provided by @zyphs21.

Haven't tried yet with the latest "trial" SDK as I'm already having enough issues with stable SDKs...

[sgallo0692]

And the mavic mini video feed shows? Are you doing anything unique to get the mini video feed?

[AaaK00]

And the mavic mini video feed shows? Are you doing anything unique to get the mini video feed?

Yeap, the video feed is working also when using Mavic Mini.

Have you checked this example project: https://github.com/dji-sdk/Mobile-SDK-iOS/tree/master/Sample%20Code/SwiftSampleCode/DJISDKSwiftDemo

By mimicking that project I was able to get the SW decoded video feed from Mini.

[sgallo0692]

I do have everything from the camera folder imported into my project and it works great for the spark so far, will keep testing for the mini if the same code works for yours

[sgallo0692]

@AaaK00 and do you turn hardware decoding off for the mini?

[AaaK00]

We are always turning the HW decoding on. For Mini it just does not stay on for long, and after that we need to create CVPixelBuffer from received VideoFrameYUV by ourself.

This is the extension I've created for the VideoFrameYUV for getting UIImage from it.

// Helper extension for getting UIImage from DJIWidgets VideoFrameYUV
extension VideoFrameYUV {

    private func createPixelBuffer () -> CVPixelBuffer? {
        var pixelBuffer: CVPixelBuffer?

        if let fastupload = self.cv_pixelbuffer_fastupload {

            // If using HW decoding, buffer can be casted directly
            pixelBuffer = unsafeBitCast(fastupload, to: CVPixelBuffer.self)
        } else {

            // Otherwise need to create buffer manually.
            let _: CVReturn = CVPixelBufferCreate(kCFAllocatorDefault, Int(self.width), Int(self.height), kCVPixelFormatType_420YpCbCr8Planar, nil, &pixelBuffer)

            guard let pixelBuffer = pixelBuffer,
                CVPixelBufferLockBaseAddress(pixelBuffer, []) == kCVReturnSuccess
                else {
                return nil
            }

            let yPlaneWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)
            let yPlaneHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)

            let uPlaneWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)
            let uPlaneHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)

            let vPlaneWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 2)
            let vPlaneHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 2)

            let yDestination = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
            memcpy(yDestination, self.luma, yPlaneWidth * yPlaneHeight)

            let uDestination = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1)
            memcpy(uDestination, self.chromaB, uPlaneWidth * uPlaneHeight)

            let vDestination = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 2)
            memcpy(vDestination, self.chromaR, vPlaneWidth * vPlaneHeight)

            CVPixelBufferUnlockBaseAddress(pixelBuffer, [])
        }
        return pixelBuffer
    }

    func createUIImage() -> UIImage? {

        if let pixelBuffer = createPixelBuffer() {
            var cgImage: CGImage?
            VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &cgImage)
            if let strongCGImage = cgImage {
                return UIImage(cgImage: strongCGImage)
            } else {
                print("Error: Unable to create CGImage from pixel buffer")
            }
        } else {
            print("Error: Unable to create pixel buffer from YUV Frame")
        }
        return nil
    }
}

[Michael-DJI]

followed with DUSDK-13560 internally