How to loop video playback and add audio to BBMetalView

[pennywise94]

Hi @Silence-GitHub,

First of all, thank you for this wonderful library. I've been successfully using it for processing images, but now I want to use it for video. I have set up a BBMetalView successfully, which shows the video I want. However, two concerns:

• I am unable to loop/repeat the video. Another user asked this question here, but the issue is closed and there is no answer/solution provided. Is there a way to loop the video, or a method to add an observer so we can know when the video playback ends, and start it again?
• My video, at this point, has no audio when it's shown through BBMetalView. I would like to have the audio played in the video. Is this possible? Should I use something other than BBMetalView?

I use BBMetalView because I want to give my users the ability to edit their video in real-time, when the video plays back to them. This works at the moment, but only within the first playback of the video (the video is usually 30 seconds long, or shorter), and there is no audio to be heard. Any ideas?

Current setup:

self.metalView = BBMetalView(frame: CGRect(x: 0, y: self.view.center.y - ((UIScreen.main.bounds.width * 1.25) / 2), width: UIScreen.main.bounds.width, height: UIScreen.main.bounds.width * 1.25))
self.view.addSubview(self.metalView)

self.videoSource = BBMetalVideoSource(url: outputURL)
self.videoSource.playWithVideoRate = true

self.videoWriter = BBMetalVideoWriter(url: newOutputURL, frameSize: BBMetalIntSize(width: 480, height: 600))

self.videoSource.audioConsumer = self.videoWriter
self.videoSource.add(consumer: self.metalView)
self.videoSource.add(consumer: self.videoWriter)

// self.videoWriter.start()
self.videoSource.start()

[Silence-GitHub]

BBMetalVideoSource has start(progress:completion:) method. To loop the video, start the video source again in the completion callback.

func start() {
    videoSource.start() { [weak self] (finish) in
        guard let self = self else { return }
        self.start()
    }
}

For audio, I am still working on it. But I am not sure how long it will take. You can use AVPlayer for audio.

[pennywise94]

Hi @Silence-GitHub, thank you for the quick reply. Will be looking into it just now, looping the video sounds logical indeed and should work. I'll try adding a separate AVPlayer and get this to work. I have an idea on how to implement it. Will start working on it now. Thanks again for the quick reply!

[pennywise94]

Is there a way to synchronize AVPlayer with the BBMetalView? For instance, is there any way I can know for sure the audio and video will be completely in sync, or should I hope for the best?

[Silence-GitHub]

Synchronizing video and audio is the problem I have not solved when working on audio player. Sorry.

[pennywise94]

That's indeed the issue I am currently facing as well. I'm looking into it as we speak, if I find a solution I'll let you know. Does BBMetalImage use a CADisplayLink or something similar to render the view? Do you know how much fps this represents or what the bitrate would be?

[Silence-GitHub]

BBMetalVideoSource dose not use CADisplayLink. If you set playWithVideoRate to true, it outputs video frames with the actual rate. However, the render view FPS is not exactly the same to the actual rate. Because the video source output frame is processed by filters and the filter process time is variable.

[pennywise94]

Do we have any way to know the "delay" caused by the filter process? E.g. the duration variable of the processing, for example? Can we do something with CACurrentMediaTime()?

[pennywise94]

I have reached out on StackOverflow to see if someone has an idea on how to sync MTKView with AVPlayer, or if someone could suggest another approach in which we can sync the audio.

[pennywise94]

You might find the comments and answers on my StackOverflow post useful, @Silence-GitHub.

Also, would it be an idea to stop drawing in the MTKView, by setting isPaused to true, and then continuing when the processing of the chosen filter is done? This might add a couple of ms delay in the filter being applied, but I think that would be more acceptable than the audio being out of sync. Or am I missing something here and is this not possible, at all? For my suggestion we would need to be able to pause the MTKView, then get notified when the filter is processed and continue drawing in the MTKView.

[pennywise94]

Another interesting reference regarding this issue: https://github.com/MetalPetal/MetalPetal/issues/40

[hoangdado]

Hi @Silence-GitHub and @pennywise94, I post an answer here. It is just a work around solution but I think you should consider this to find the general solution.

[pennywise94]

The solution provided by @hoangdado worked for me. @Silence-GitHub I feel like this should be implemented in the project, as it works smoothly now for audio playback when showing video.