RTMP Live Streaming - VideoPreviewer distortion + crashing

[sgallo0692]

After reading many issues/stackoverflow articles/DJI forums, I've finally got streaming working somewhat reliably. However, actually getting the video feed to stabilize is becoming challenging. I'm facing two major issues testing with a DJI Spark:

1. The DJIVideoPreviewer seems to crash often. Specifically when I begin streaming. It seems when I have streaming running, when the video feed gets intensive (i.e. not inside, staring at a wall) it crashes after a few seconds. Is there a way to be throttling or managing memory to prevent this? Something is getting overloaded and I'm not sure if it's CPU or memory or what but it seems I need to be doing something more efficient here.

2. The video feed often appears heavily distorted which seems linked to the videoFeed function but I'm not entirely sure what the cause is. This occasionally just doesn't happen, but most often when I open the app my feed looks like this:

This is most of my code wrapped in a UIViewController class that I call to setup the fpv view

Video processing:

extension FPVView: DJIVideoFeedListener, DJICameraDelegate {

    func videoFeed(_ videoFeed: DJIVideoFeed, didUpdateVideoData rawData: Data) {

        let videoData = rawData as NSData
        let videoBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: videoData.length)

        videoData.getBytes(videoBuffer, length: videoData.length)

        DJIVideoPreviewer.instance().push(videoBuffer, length: Int32(videoData.length))
        videoBuffer.deallocate()
    }

    func camera(_ camera: DJICamera, didUpdate cameraState: DJICameraSystemState) {
        self.isRecording = cameraState.isRecording
    }
}

extension FPVView: VideoFrameProcessor {

    func videoProcessorEnabled() -> Bool {
        return true
    }

    func videoProcessFrame(_ frame: UnsafeMutablePointer<VideoFrameYUV>!) {
//        let resolution = CGSize(width: CGFloat(frame.pointee.width), height: CGFloat(frame.pointee.height))
        if frame.pointee.cv_pixelbuffer_fastupload != nil {
            //  cv_pixelbuffer_fastupload to CVPixelBuffer
            let cvBuf = unsafeBitCast(frame.pointee.cv_pixelbuffer_fastupload, to: CVPixelBuffer.self)
            pixelBuffer = cvBuf
            print("pushed video1")

        } else {
            // create CVPixelBuffer by your own, createPixelBuffer() is an extension function for VideoFrameYUV
            pixelBuffer = frame.pointee.createPixelBuffer()
//            guard let cvBuf = pixelBuffer else { return }
            print("pushed video2")
        }
    }
}

extension VideoFrameYUV {
    func createPixelBuffer() -> CVPixelBuffer? {
        var initialPixelBuffer: CVPixelBuffer?
        print("---creatPixel: \(self.width), \(self.height)")
        let _: CVReturn = CVPixelBufferCreate(kCFAllocatorDefault, Int(self.width), Int(self.height), kCVPixelFormatType_420YpCbCr8Planar, nil, &initialPixelBuffer)

        guard let pixelBuffer = initialPixelBuffer
            , CVPixelBufferLockBaseAddress(pixelBuffer, []) == kCVReturnSuccess
            else {
                return nil
        }

        let yPlaneWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)
        let yPlaneHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)

        let uPlaneWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)
        let uPlaneHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)

        let vPlaneWidth = CVPixelBufferGetWidthOfPlane(pixelBuffer, 2)
        let vPlaneHeight = CVPixelBufferGetHeightOfPlane(pixelBuffer, 2)

        let yDestination = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0)
        memcpy(yDestination, self.luma, yPlaneWidth * yPlaneHeight)

        let uDestination = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1)
        memcpy(uDestination, self.chromaB, uPlaneWidth * uPlaneHeight)

        let vDestination = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 2)
        memcpy(vDestination, self.chromaR, vPlaneWidth * vPlaneHeight)

        CVPixelBufferUnlockBaseAddress(pixelBuffer, [])

        return pixelBuffer
    }
}

Video previewer + stream setup

    var fpvController: DUXFPVViewController?
    var pixelBuffer: CVPixelBuffer?
    var streamKey: String?
    var isStreaming: Bool?

    var isRecording: Bool?

    override func viewDidLoad() {
        super.viewDidLoad()

        if let camera = fetchCamera() {
            camera.delegate = self
        }

        self.setupVideoPreviewer()
        self.fpvController = DUXFPVViewController()
        self.fpvController?.fpvView?.showCameraDisplayName = false
        self.addChild(self.fpvController!)
    }

    override func viewDidDisappear(_ animated: Bool) {
        let RTMPMuxer = DJIRtmpMuxer.sharedInstance()
        guard RTMPMuxer != nil else {
            print("rtmp muxer not ready")
            return
        }
        RTMPMuxer?.stop()

        DJISDKManager.videoFeeder()?.primaryVideoFeed.remove(self)
        DJIVideoPreviewer.instance().unSetView()
        DJIVideoPreviewer.instance().close()
    }

    func setupVideoPreviewer() {

            // So we can try and grab video frames
            DJIVideoPreviewer.instance().type = .autoAdapt
            DJIVideoPreviewer.instance().enableHardwareDecode = true
            DJIVideoPreviewer.instance()?.registFrameProcessor(self)
            DJIVideoPreviewer.instance()?.enableFastUpload = true
            DJIVideoPreviewer.instance().setView(self.view)

            let product = DJISDKManager.product();

            //Use "SecondaryVideoFeed" if the DJI Product is A3, N3, Matrice 600, or Matrice 600 Pro, otherwise, use "primaryVideoFeed".
            if ((product?.model == DJIAircraftModelNameA3)
                || (product?.model == DJIAircraftModelNameN3)
                || (product?.model == DJIAircraftModelNameMatrice600)
                || (product?.model == DJIAircraftModelNameMatrice600Pro)){
                DJISDKManager.videoFeeder()?.secondaryVideoFeed.add(self, with: nil)
            }else{
                DJISDKManager.videoFeeder()?.primaryVideoFeed.add(self, with: nil)
            }

            DJIVideoPreviewer.instance().start()
    }

    func fetchCamera() -> DJICamera? {
        let product = DJISDKManager.product()

        if (product == nil) {
            return nil
        }

        if (product!.isKind(of: DJIAircraft.self)) {
            return (product as! DJIAircraft).camera
        } else if (product!.isKind(of: DJIHandheld.self)) {
            return (product as! DJIHandheld).camera
        }

        return nil
    }

    func startStream() {
        if(streamKey != nil) {
            let RTMPMuxer = DJIRtmpMuxer.sharedInstance()
            guard RTMPMuxer != nil else {
                print("Error: rtmp muxer not ready")
                return
            }
            RTMPMuxer?.setupVideoPreviewer(DJIVideoPreviewer.instance())
            RTMPMuxer?.serverURL = "rtmp://global-live.mux.com:5222/app/\(streamKey!)"
            RTMPMuxer?.enabled = true
            RTMPMuxer?.enableAudio = true
            RTMPMuxer?.retryCount = 3
            RTMPMuxer?.start()
        } else {
            print("Error: stream key is nil")
        }
    }

[sgallo0692]

@dji-dev do you have any engineers particularly skilled in this area? Thanks!

[dji-dev]

Agent comment from Luce Luo in Zendesk ticket #41888:

Dear Customer,

Thank you for contacting DJI. This is the DJI SDK support team. We are checking your issue on GitHub #403. https://github.com/dji-sdk/Mobile-SDK-iOS/issues/403

If the problem still, please contact us again.

Thanks,

Luce Luo DJI Developer Support

[sgallo0692]

@dji-dev i still haven't figured this out and still need help

[sgallo0692]

Hi,

I still need help on this github issue. I believe i’m following fpv examples correctly but is crashing still and the feed looks distorted as my screenshot shows

-Steve

On Dec 8, 2020, at 9:26 PM, DJI notifications@github.com wrote:

 Agent comment from Luce Luo in Zendesk ticket #41888:

Dear Customer,

Thank you for contacting DJI. This is the DJI SDK support team. We are checking your issue on GitHub #403.

403

If the problem still, please contact us again.

Thanks,

Luce Luo DJI Developer Support

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[Matheus193dn]

@sgallo0692 hey, I can live stream with my Matrice 300 RTK using DJIMuxer. You need to import VideoPreviewerAdapater & DecodeLogic in DJI Sample to your project.

[sgallo0692]

@Matheus193dn really appreciate the reply with a code sample - any crashing or quality issues with the stream?

[Matheus193dn]

@sgallo0692 It works fine. I only have issue with delaying time(still better than Haishinkit & LFLiveKit)

[sgallo0692]

awesome, I'll try that. I'm looking around for DecodeLogic and having trouble finding it - do you have a GitHub link to where that exists in the sample project? is it this: SwiftSampleCode/DJISDKSwiftDemo/Camera/DecodeImageCalibrateLogic.swift

[Matheus193dn]

@sgallo0692 exactly

[sgallo0692]

confirming this resolved the issue and worked perfectly. thanks again

[sgallo0692]

@Matheus193dn sorry to bother but not sure who else to ask at this point

Using your code above, do you have any stability issues? Any tips for retries/stabilizing the rtmp stream? Trying to make it robust and restart when the connection is spotty or stops working

[Matheus193dn]

@sgallo0692 Muxer does have delegate protocol, you can observe the livestream status if it has errors. You can call rtmpMuxer.start() again. Also, it does provide retryCount, pass the number of times you want to retry(my sample code is 3).

[sgallo0692]

I'll start there, i think i was calling all of your code again instead of just rtmpmuxer.start()

Appreciate the suggestions

[Matheus193dn]

@sgallo0692 you can take a look at DJI sample here. They used DJIRtmpMuxerStateUpdateDelegate to observe rmtpMuxerstate.

https://github.com/dji-sdk/Mobile-SDK-iOS/blob/master/Sample%20Code/ObjcSampleCode/DJISdkDemo/Demo/Liveview/VideoLiveStreamingViewController.m

[sgallo0692]

Tried implementing this with no luck printing the status changes yet, i may have messed up translating to swift but going to keep working on getting this too.

Seems critical to relaunching the stream if it's broken