Camera() has buffering issue

[gemblerz]

Camera() utilizes OpenCV's VideoCapture to get stream from camera device. The stream could be either from USB or Network via RTSP protocol. When VideoCapture opens up a stream we need to keep pulling frames from the stream to get the latest frame. If not, it would give a buffered frame with the wrong timestamp to users.

[gemblerz]

Current use cases of the Camera() are,

NOTE: the app is checked if it possibly falls into the buffering problem

• ( ) cloud motion app: opens a new with Camera() as cap and snapshot() wherever the app needs to grab a frame.
• ( ) smoke detection app: uses snapshot() to get one image
• (X) water depth app: uses snapshot() with the with Camera() outside the loop, a potential cause of the buffering
• (X) motion detection app: uses stream() but not calling it as fast as the stream's FPS, possibly causing the buffering issue. the app would stop after publishing measurements for a given number
• ( ) traffic state app: uses python ffmpeg to save a video clip, so not using our Camera()
• (X) object counter app: uses stream() with the with Camera() outside the loop, a potential cause of the buffering when running continuously
• (X) surface water app: uses snapshot() but not calling it as fast as the stream's FPS, possibly causing the buffering issue.
• ( ) motion analysis app: uses python ffmpeg to save a video clip, so not using our Camera()
• ( ) cloud cover app: uses snapshot() without the with Camera() statement, which makes the device to be opened only on snapshot()

[gemblerz]

After reviewing the apps, it seems there are 3 use cases,

1. Apps need to obtain a frame to process the frame. No relationship between frames so obtaining the next frame can happen whenever needed Examples: cloud motion, water depth, surface water, and cloud cover

2. Some apps need to process frames as fast as they can. But, the processing time can't keep up with the camera FPS. So, they want to obtain the next frame, which should be the latest, whenever they are ready to process Examples: cloud motion, smoke detection, motion detection, Nico's object detection and tracking, and object counter

3. Some other apps need a video clip that captures frames at the camera FPS and process it Examples: traffic state and motion analysis

With the uses cases in mind, I suggest,

• use snapshot() as is without the with Camera(device) as cap to cover the use case 1. using the with statement can possibly cause the buffering issue still because the statement opens the stream, but not draining the buffer.

• repurpose stream() and add a thread in the function to support the use case 2

  with Camera(device) as cap:
  for sample in cap.stream(): # this always gives the latest frame
      # do processing

• add record() that returns a file path after the recording. this covers the use case 3

  # first record and store frames into a file
  with Camera(device) as cap:
  file_path, error = cap.record(duration=60) 
  # then read the record file to process
  with Camera(file_path) as file:
  # for sample in file.stream(): # this is incorrect use for this proposal :(
  frame = file.snapshot()
  # do processing