Screenshots in SDK

[Shagans982]

Is there anything preventing screenshots to be taken on the TVIVideoView views? Everything I take comes back black. Capturing views other than these work fine.

[ceaglest]

Hey @Shagans982,

I'm not sure, TVIVideoView is rendering with Metal but it doesn't do anything else to protect the content or prevent it from being captured.

There are some alternatives. You could write your own TVIVideoRenderer if you want still frames from each VideoTrack. If you only care about the local feed, maybe a TVICameraPreviewView would work instead?

Best, Chris

[Shagans982]

I am assuming I just need to capture the buffer from renderFrame and convert that into a UIImage. When is renderFrame hit bc I put this in place but nothing seems to be coming through while a video is live.

final class RemoteCameraRenderer: TVIVideoView {
    var screenshot: UIImage?

    override func renderFrame(_ frame: TVIVideoFrame) {
        super.renderFrame(frame)
        let image = CIImage(cvImageBuffer: frame.imageBuffer, options: nil)
        let context = CIContext()
        let cgiImage = context.createCGImage(image, from: image.extent)
        screenshot = UIImage(cgImage: cgiImage!)
    }
}

[Shagans982]

I have also tried obtaining a MLTTexture from Metal to get a screenshot but neither is working.

[Shagans982]

any updates on this?

[andschdk]

Hi @Shagans982

Try implementing a VideoRenderer rather than subclassing TVIVideoView.

Note: below code is using latest version - 'namespace' is no longer TVI

Call captureFrame(completion: @escaping (UIImage?) -> Void) to get the next available frame.

class FrameCaptureRenderer: NSObject, VideoRenderer {
    private var captureNextFrame = false
    private var completions: [(UIImage?) -> Void] = []

    public func updateVideoSize(_ videoSize: CMVideoDimensions, orientation: VideoOrientation) { }

    func renderFrame(_ frame: VideoFrame) {
        if !captureNextFrame {
            return
        }

        captureNextFrame = false

        let imageBuffer = frame.imageBuffer
        let ciImage = CIImage(cvImageBuffer: imageBuffer, options: nil)
        let image = UIImage(ciImage: ciImage)
        completions.forEach { $0(image) }
        completions = []
    }

    func captureFrame(completion: @escaping (UIImage?) -> Void) {
        completions.append(completion)
        captureNextFrame = true
    }
}

[Shagans982]

Correct me if I am wrong but isn't TVIVideoView backed by TVIVideoRenderer? Trying to understand how I VideoRenderer without the view.

[Shagans982]

Just to make you aware, my TVIVideoView is created from the Storyboard which won't allow it to be a class type of NSObject.

[ceaglest]

Correct me if I am wrong but isn't TVIVideoView backed by TVIVideoRenderer? Trying to understand how I VideoRenderer without the view.

@Shagans982 TVIVideoView implements TVIVideoRenderer. You can write your own renderer, and the code snippet from @andschdk looks correct to me.

[Shagans982]

I have a view object in the storyboard that needs a custom class. Currently its TVIVideoView (which implements TVIVideoRenderer), the code above is an NSObject thus I cannot add as a class to the storyboard view. It needs to be a TVIVideoView. However, when I do that none of the methods are hit.

I am assuming the only way to write a renderer is by subclassing?

[Shagans982]

Hi @Shagans982

Try implementing a VideoRenderer rather than subclassing TVIVideoView.

Note: below code is using latest version - 'namespace' is no longer TVI

Call captureFrame(completion: @escaping (UIImage?) -> Void) to get the next available frame.

class FrameCaptureRenderer: NSObject, VideoRenderer {
    private var captureNextFrame = false
    private var completions: [(UIImage?) -> Void] = []

    public func updateVideoSize(_ videoSize: CMVideoDimensions, orientation: VideoOrientation) { }

    func renderFrame(_ frame: VideoFrame) {
        if !captureNextFrame {
            return
        }

        captureNextFrame = false

        let imageBuffer = frame.imageBuffer
        let ciImage = CIImage(cvImageBuffer: imageBuffer, options: nil)
        let image = UIImage(ciImage: ciImage)
        completions.forEach { $0(image) }
        completions = []
    }

    func captureFrame(completion: @escaping (UIImage?) -> Void) {
        completions.append(completion)
        captureNextFrame = true
    }
}

Maybe I am missing something here but where is this class instantiated and added as a renderer? The only places I see renderers being added is from the TYVIVideoView. Am I missing something?

[andschdk]

Initialization depends on your setup, but it could be in the view controller similar to the QuickStart sample app. Renderers are added to video tracks. See TVIVideoTrack. You will most likely end up adding 2 renderers to your video track:

• TVIVideoView
• FrameCaptureRenderer (the custom renderer I posted)

Hope it makes sense.

[piyushtank]

Closing the ticket as the current solution is discussed in this thread. Please re-open if you have any questions.

[kajaldabrai]

Hi @Shagans982

Try implementing a VideoRenderer rather than subclassing TVIVideoView.

Note: below code is using latest version - 'namespace' is no longer TVI

Call captureFrame(completion: @escaping (UIImage?) -> Void) to get the next available frame.

class FrameCaptureRenderer: NSObject, VideoRenderer {
    private var captureNextFrame = false
    private var completions: [(UIImage?) -> Void] = []

    public func updateVideoSize(_ videoSize: CMVideoDimensions, orientation: VideoOrientation) { }

    func renderFrame(_ frame: VideoFrame) {
        if !captureNextFrame {
            return
        }

        captureNextFrame = false

        let imageBuffer = frame.imageBuffer
        let ciImage = CIImage(cvImageBuffer: imageBuffer, options: nil)
        let image = UIImage(ciImage: ciImage)
        completions.forEach { $0(image) }
        completions = []
    }

    func captureFrame(completion: @escaping (UIImage?) -> Void) {
        completions.append(completion)
        captureNextFrame = true
    }
}

Where I need to assign this new renderer called FrameCaptureRenderer? To a Video Track?

[kajaldabrai]

Initialization depends on your setup, but it could be in the view controller similar to the QuickStart sample app. Renderers are added to video tracks. See TVIVideoTrack. You will most likely end up adding 2 renderers to your video track:

• TVIVideoView
• FrameCaptureRenderer (the custom renderer I posted)

Hope it makes sense.

How to achieve this? where I can add FrameCaptureRenderer?

[andschdk]

@kajaldabrai You add the FrameCaptureRenderer as a renderer on the TVIVideoTrack using addRenderer(...) https://twilio.github.io/twilio-video-ios/docs/latest/Classes/TVIVideoTrack.html#//api/name/addRenderer:

[Byronium]

Hi @Shagans982

Try implementing a VideoRenderer rather than subclassing TVIVideoView.

Note: below code is using latest version - 'namespace' is no longer TVI

Call captureFrame(completion: @escaping (UIImage?) -> Void) to get the next available frame.

class FrameCaptureRenderer: NSObject, VideoRenderer {
    private var captureNextFrame = false
    private var completions: [(UIImage?) -> Void] = []

    public func updateVideoSize(_ videoSize: CMVideoDimensions, orientation: VideoOrientation) { }

    func renderFrame(_ frame: VideoFrame) {
        if !captureNextFrame {
            return
        }

        captureNextFrame = false

        let imageBuffer = frame.imageBuffer
        let ciImage = CIImage(cvImageBuffer: imageBuffer, options: nil)
        let image = UIImage(ciImage: ciImage)
        completions.forEach { $0(image) }
        completions = []
    }

    func captureFrame(completion: @escaping (UIImage?) -> Void) {
        completions.append(completion)
        captureNextFrame = true
    }
}

This VideoRenderer pattern worked like a charm! Would be great to mention this in docs or have a code example in this repo.

For what it's worth I was a bit confused by the completions array. I'm presuming this is an array of callbacks, and is such that if captureFrame is called many times in rapid succession, the callback can be called on each frame.

For my use case, captureFrame wouldn't be called in rapid succession, so I used the following simpler implementation:

class FrameCaptureRenderer: NSObject, VideoRenderer {
    private var captureNextFrame = false
    private var onFrameCaptured: (UIImage?) -> Void

    init(onFrameCaptured: @escaping (UIImage?) -> Void) {
        self.onFrameCaptured = onFrameCaptured
    }

    public func updateVideoSize(_ videoSize: CMVideoDimensions, orientation: VideoOrientation) { }

    func renderFrame(_ frame: VideoFrame) {
        // by default, do nothing with the frame
        if !captureNextFrame {
            return
        }

        captureNextFrame = false

        let imageBuffer = frame.imageBuffer
        let ciImage = CIImage(cvImageBuffer: imageBuffer, options: nil)
        let image = UIImage(ciImage: ciImage)
        onFrameCaptured(image)
    }

    func captureFrame() {
        // set captureNextFrame to true so that the next call to renderFrame will capture the frame
        captureNextFrame = true
    }
}

  // To initialize
  frameCaptureRenderer = FrameCaptureRenderer(onFrameCaptured: <insert callback here>)
  localVideoTrack?.addRenderer(frameCaptureRenderer!)
  ...
  // To call:
  frameCaptureRenderer.captureFrame()