search

hollance / YOLO-CoreML-MPSNNGraph

Tiny YOLO for iOS implemented using CoreML but also using the new MPS graph API.
MIT License
929 stars 251 forks source link

Question regarding semaphore concept and frame dropping #44

Closed strobmax closed 5 years ago

strobmax commented 5 years ago

Hi,

I noticed you use the following concept in most of your projects:

let semaphore = DispatchSemaphore(value: 1)

extension ViewController: VideoCaptureDelegate {
  func videoCapture(_ capture: VideoCapture, didCaptureVideoFrame pixelBuffer: CVPixelBuffer?, timestamp: CMTime) {

      semaphore.wait()

      runCoreMLRequest(on: pixelBuffer) {
           semaphore.signal()
      }
}

You do this with the intention of dropping frames and this works, because you set AVCaptureVideoDataOutput.alwaysDiscardsLateVideoFrames = true.

However - and here comes my question - since DispatchSemaphore is a FIFO queue, won't you start the next CoreML request with the --oldest-- frame instead of the latest? I suspect that this is because videocaptures drops all newer frames. After the semaphore get's free, the frame that got emitted as the first one after the CoreML request started will be taken for the next request. Maybe here an exemplary sequence explaining what I mean:

frame 1 arrives CoreML request starts with frame 1 frame 2 arrives and gets stuck in wait() frame 3 gets dropped because previous frame processing stuck frame 4 gets dropped because previous frame processing stuck CoreML request finishes and calls semaphore.signal() the frame that is stuck in wait() (frame 2) gets unblocked and runCoreMLRequest is started CoreML request starts with frame 2

This would introduce an unnecessary delay of the duration of a signal NN forward pass. Is that correct?

hollance commented 5 years ago

You are correct. :-) The video thread blocks on an older frame and any newer frames get dropped. So when the thread is freed up again, you get an older frame.

I haven't really investigated a solution that involves always using the very latest frame. This would require a custom queue of some kind, I guess.

However, if you have a model that can't really keep up with the video feed anyway, then you probably shouldn't try to use it at 30 FPS. ;-)

strobmax commented 5 years ago

Thanks for your quick response and explanation :)