Silence-GitHub / BBMetalImage

A high performance Swift library for GPU-accelerated image/video processing based on Metal.
MIT License
989 stars 126 forks source link

Capture video & playback in same metalView issues #29

Closed pennywise94 closed 3 years ago

pennywise94 commented 4 years ago

I am trying to record a video by start pressing on a record button, and when releasing I want to play back the video I just recorded in the same metalView. Below is my code, but I am facing a few hiccups:

  1. The first second, when I start recording, the metalView freezes, so the first few frames are not smoothly recorded. This is also noticeable in the output.
  2. As soon as I end the recording, I get the message "Asset writer or audio input is not ready for writing this frame" a couple of times (+- 30 times).

Any ideas on how to improve this code and how I can achieve the result I want? Don't mind the AudioPlayer stuff. I also want to play the audio of the video I just recorded.

    @objc func videoAction(_ sender: UILongPressGestureRecognizer) {
        if(sender.state == .began) {
            print("began")
            startTime = CACurrentMediaTime()
            isRecording = true
            recordTimer = Timer.scheduledTimer(timeInterval: 0.1, target: self, selector: #selector(updateRecordProgress), userInfo: self, repeats: true)
            videoWriter.start { (type) in
            }
        } else if(sender.state == .ended) {
            print("ended")
            self.recordTimer?.invalidate()
            if(isRecording) {
                isRecording = false
                let videoURL = self.outputURL
                videoWriter.finish {
                    DispatchQueue.main.async { [weak self] in
                        guard let self = self else { return }
                        self.camera.removeAllConsumers()
                        self.videoSource = BBMetalVideoSource(url: self.outputURL)
                        self.videoSource.playWithVideoRate = true
                        self.videoSource.start(progress: { (frameTime) in
//                          print(frameTime)
                        }) { [weak self] (_) in
                            guard let self = self else { return }
                            self.videoSource.add(consumer: self.metalView!)
                            self.audioItem = AVPlayerItem(url: self.outputURL)
                            self.audioPlayer = AVPlayer(playerItem: self.audioItem)
                            self.audioPlayer.addObserver(self, forKeyPath: "status", options: [.old, .new], context: nil)
                            self.playerLayer = AVPlayerLayer(player: self.audioPlayer)
                            self.view.layer.addSublayer(self.playerLayer!)
                            self.playerLayer?.frame = CGRect(x: 0, y: 0, width: 0, height: 0)
                            self.playerLayer?.backgroundColor = UIColor.black.cgColor
                            print("done")
                        }
                    }
                }
            }
        }
    }
pennywise94 commented 4 years ago

@Silence-GitHub any idea how I could accomplish the behavior I want, within one and the same metalView? So I can record a video and play back almost immediately in the same metalView?

Silence-GitHub commented 4 years ago

For question 1: The first time start recording, some frames freeze. The next time works well. It seems only the first time start recording (after launching the app) has the problem. And it seems this happens only when launching the app by Xcode. If I disconnect iPhone from Xcode, and launch the app by touching iPhone, it works well. If the problem happens only when launching the app by Xcode, you can ignore it. But I am not sure since I don't have more device to test.

For question 2: You can ignore the log. There is the same log in the demo project. Finishing recording takes time. And during this time, some frames will be sent to the video writer which can not accept more frames.