Capture the current frame in composed player item

[shacharudi]

First of all, thank you very much for creating this framework. It's great and I really appreciate all the time and effort you put in to it 🙏🙏🙏

I'm creating a video editing app and I'm trying to apply a filter and then edit the filter configuration without reloading the video composition and updating the player item.

Basically I have a AVPlayer that has a paused video (at readyToPlay status) and I'm allowing the user to add a filter (using the Timeline.passingThroughVideoCompositionProvider object). This works great and I can see my filter working as expected.

Now, I would like to allow the user to change the configuration of the filter (for example the radius or the center) but I don't want to reload the composition after every change. I'd like it to happen in real time and reloading the composition would take too long and the UX won't be great.

My approach is to:

1. Disable the filter
2. Get the current frame of the video composition as an UIImage (Original Image)
3. Apply the filter on the UIImage that I captured
4. Add a UIImageView with the filtered image on top of the video player
5. When the user change the filter config, apply it to the Original Image
6. When the user is done, remove the UIImageView, update the Timeline.passingThroughVideoCompositionProvider and reload the composition

---

First Question Is that a proper approach? Do you know a better approach I can take?

Second Question How can I capture a UIImage of a specific time in the timeline?

---

My issue with this approach is with action number 2 - getting the current frame of the video composition I tried many approaches to capture the image but nothing worked. The results I'm getting are either a black/transparent image or an image of a video track without any overlays that exists in the same currentTime.

Here are some code examples that returns an empty image:

public func getCurrentFrameScreenShot() -> UIImage? {
    guard let playerLayer = self.playerLayer else { return nil }
    UIGraphicsBeginImageContextWithOptions(playerLayer.frame.size, playerLayer.isOpaque, 0)
    playerLayer.render(in: UIGraphicsGetCurrentContext()!)
    let outputImage = UIGraphicsGetImageFromCurrentImageContext()
    UIGraphicsEndImageContext()
    return outputImage!
}

public func getCurrentFrameScreenShot() -> UIImage? {
    return UIGraphicsImageRenderer(size: bounds.size).image { _ in
      drawHierarchy(in: CGRect(origin: .zero, size: bounds.size), afterScreenUpdates: true)
    }
}

public func getCurrentFrameScreenShot() -> UIImage? {
    guard let player = self.player else { return nil }
    guard let currentItem = player.currentItem else { return nil }

    let videoOutput = AVPlayerItemVideoOutput(pixelBufferAttributes: [String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_32BGRA])
    self.videoOutput = videoOutput
    self.player?.currentItem?.add(videoOutput)

    let currentTime = currentItem.currentTime()
    let buffer = self.videoOutput?.copyPixelBuffer(forItemTime: currentTime, itemTimeForDisplay: nil)
    let ciImage = CIImage(cvPixelBuffer: buffer!)
    let image = UIImage(ciImage: ciImage)

    return image
}

// This function works but only capture a video track. Not capturing any image tracks or overlays
public func getCurrentFrameScreenShot() -> UIImage? {
    guard let player = self.player else { return nil }
    guard let asset = player.currentItem?.asset else { return nil }

    let imageGenerator = AVAssetImageGenerator(asset: asset)
    imageGenerator.requestedTimeToleranceAfter = CMTime.zero
    imageGenerator.requestedTimeToleranceBefore = CMTime.zero

    guard let image = try? imageGenerator.copyCGImage(at: player.currentTime(), actualTime: nil) else {
        return nil
    }

    return UIImage(cgImage: image)
}

I know this is a long question so thank you for reading it. I would appreciate any feedback/response. 🙏

[shacharudi]

I spent a few (long) hours trying to solve this. A few minutes after posting this issue I found out I can use the CompositionGenerator.buildImageGenerator() function. 🤦‍♂️

I'll leave this here in case anyone else is looking for something similar

public func getCurrentFrameScreenShot() -> UIImage? {
    guard let compositionGenerator = compositionGenerator else {
        return nil
    }

    guard let player = self.player else { return nil }

    let imageGenerator = compositionGenerator.buildImageGenerator()
    imageGenerator.requestedTimeToleranceAfter = CMTime.zero
    imageGenerator.requestedTimeToleranceBefore = CMTime.zero

    guard let image = try? imageGenerator.copyCGImage(at: player.currentTime(), actualTime: nil) else {
        return nil
    }

    return UIImage(cgImage: image)
}