Possible frame drops

[gwappa]

The number of strobe triggers do not often match with that of the resulting frames in the saved video.

Some of the most recent figures:

Frame rate (FPS)	# strobe triggers	# video frames	Difference
30	111319	111315	4
30	109556	109552	4
30	83774	83770	4
160	592876	592863	13
160	583540	583531	9
160	445965	445952	13

Based on the discussion in this issue https://github.com/TheImagingSource/IC-Imaging-Control-Samples/issues/37 , frame drops must be strictly avoided by minimizing the overhead taken to process individual frames (i.e. during callbacks).

Possible bottlenecks (those I can think of currently):

• The multi-process environment:
  • Spawning of a new ffmpeg process
  • Writing bytes of frames through a pipe
• The use of python GIL at any of the above steps

There are two possibilities:

1. Use an FIFO to buffer frames before being stored.
2. Directly link ffmpeg-related libraries, cf.
   • Cython-based bindings (PyAV): https://pyav.org/docs/stable/
   • Build a binding by myself: https://github.com/leandromoreira/ffmpeg-libav-tutorial

[gwappa]

migrated to use PyAV.

What was observed during testing 1f77fc3 :

1. the storage procedures freezes from time to time (and I had to re-start GRAB every time).
2. there remains to be (seemingly) frame drops.

Possible causes: the additional frame-copy step. Some rough checks on a decent CPU tell that a single-frame copy() (640 x 480 RGB24) can take ~3 ms on average! On the other hand, just copying the content took ~0.3 ms/frame, suggesting most of the latency comes from memory allocation.

Temporary solutions:

1. Use a double-buffered scheme so that both callback and the encoding contexts do not share a single mutex in common (322709a).
2. Create buffers of frame objects in BufferThread so that array allocation should not occur in the callback context (3fa2c1f).

[gwappa]

tested 5226735, and found the results to be even worse than when I used bare ffmpeg...

the (relatively heavy) use of python in the callback context seems to be the culprit. It may be necessary to refine the encoder code based on PyAV, and migrate completely to Cython-based.

The use of bare ffmpeg (possibly with buffers?) could be at least better than PyAV-based code. I will try this approach for the time being.

[gwappa]

1e42336 back to the original figures after migrating to ffmpeg pipe + buffering.

In theory, further steps may be taken as:

1. Write a Cython module to buffer frames without bothering the GIL.
2. Write a Cython module to write to a pipe without bothering the GIL.
3. Write a Cython module to directly call e.g. libavcodec without bothering the GIL.

But I have no idea how long it could take for me to implement them...

[gwappa]

come to think of it, buffering frames as soon as they are acquired (in addition to buffering at the storage step) would be even better solution.

need to modify ks-tisgrabber then...

[gwappa]

as of 2b1e938

• 10255 / 10267 (0.16 frame/s) at 200 FPS 640x480
• 3017 / 3020 at (0.01 frame/s) 10 FPS 640x480
• 3096 / 3099 (0.06 frame/s) at 60 FPS 640x480
• 3187 / 3195 (0.15 frame/s) at 60 FPS 320x240
• 8055 / 8066 (0.23 frame/s) at 169 FPS 320x240

[gwappa]

it does not seem like the limit of GPU encoding but rather an error in task prioritization (which I probably do not have much control on).

[gwappa]

Now everything is clear:

• If one uses internal triggering (aka free-running mode), the camera starts its acquisition process immediately after the prepareLive() call. This means (1) one will always miss several "frame triggers" up to the point startLive() is called, and (2) on e can miss a frame or two after calling stopLive(), as it can prevent the camera from sending the incoming frame(s).
• The above feature is 'the expected behavior' for ImagingSource cameras, so there is nothing one can help; use software or external-hardware triggering if you really care about frame drops.

[gwappa]

The problem now is to reworking with the system of trigger modes... (see #22 )