Knowing when callback fails

[abisi]

Hello,

We're trying to run 2 TIS cameras continuously with FPS 200Hz, Y800 (720x540) [Binning 2x] format for durations up to ~2h. We need to know when each frame is acquired (i.e. time of exposure) in order to sync these exposure times with other data streams. To do this, we operate cameras at > 400 Hz and perform "Software Trigger" at 200 Hz. The resulting frame is added to a cv2.VideoWriter object to save a .avi file. The issue is that we obtain more exposure times (out pulses from the camera pin) than frames in our video .avi object. As order of magnitude: a few tens of frames for minutes long videos. We're trying to match the frames to individual pulses. Here are pieces of code that do what I just said.

Callback userdata and function:

class CallbackUserdata(C.Structure):
    """ User data passed to the callback function. """
    def __init__(self, camera_index):
        self.camera_index = camera_index
        self.trigger_times = []
        self.callback_times = []
        self.framenumbers = []
        self.saved_times = []
        self.image_description = None
        self.vid_out = None

def frame_callback(hGrabber, pBuffer, framenumber, pData):

    # Get image description values
    Width = pData.image_description[0]
    Height = pData.image_description[1]
    BitsPerPixel = pData.image_description[2]
    colorformat = pData.image_description[3]
#
    # Calculate the buffer size, write frame to video object
    bpp = int(BitsPerPixel / 8.0)
    buffer_size = Width * Height * bpp

    # Log callbacks
    pData.callback_times.append(time.time())
    if buffer_size > 0:
        image = C.cast(pBuffer,
                            C.POINTER(
                                C.c_ubyte * buffer_size))
#
        frame = np.ndarray(buffer=image.contents,
                                 dtype=np.uint8, #Y800
                                 shape=(Height,
                                        Width,
                                        bpp))

        pData.vid_out.write(cv2.flip(frame,0))
        pData.saved_times.append(time.time())
        pData.framenumbers.append(framenumber)
    return

FRAMEREADYCALLBACK = C.CFUNCTYPE(C.c_void_p, C.c_int, C.POINTER(C.c_ubyte), C.c_ulong, C.py_object)  
callback_function_ptr = FRAMEREADYCALLBACK(frame_callback)

Camera initialization and software trigger:

    def __init__(self, cam_num=0, rotate=None, crop=None, exposure=None):
        '''
        Params
        ------
        cam_num = int; camera number (int)
            default = 0
        crop = dict; contains ints named top, left, height, width for cropping
            default = None, uses default parameters specific to camera
        '''

        self.cam_num = cam_num
        self.rotate = rotate if rotate is not None else cam_details[str(self.cam_num)]['rotate']
        self.crop = crop if crop is not None else cam_details[str(self.cam_num)]['crop']
        self.exposure = exposure if exposure is not None else cam_details[str(self.cam_num)]['exposure']

        self.user_data = CallbackUserdata(cam_num)

        self.cam = ic.TIS_CAM()
        self.cam.open(self.cam.GetDevices()[cam_num].decode())

        # Load device state from file
        if self.cam.IsDevValid():
            try:
                file_name = 'configs/{}_config.xml'.format(cam_details[str(self.cam_num)]['name'])
            except:
                print('Could not find a device configuration state .xml file for this camera. Please check camera names.')

            self.cam.LoadDeviceStateFromFile(file_name)
            print('Loaded device state: {}'.format(file_name))

  def software_trigger(self):
        self.user_data.image_description = self.cam.GetImageDescription()
        self.user_data.trigger_times.append(time.time())
        self.cam.PropertyOnePush('Trigger', 'Software Trigger')
        return

    def set_frame_ready_callback(self):
        self.cam.SetFrameReadyCallback(callback_function_ptr, self.user_data)

Saving frames at desired frame rate using trigger mode:

def record_on_thread(self, num):
        fps = int(self.fps.get())
        start_time = time.time()
        next_frame = start_time
        try:
            while self.record_on.get():
                if time.time() >= next_frame:
                    self.frame_times[num].append(time.time())
                    self.cam[num].software_trigger()
                    self.post_trigger_times.append(time.time())
                    next_frame = max(next_frame + 1.0/fps, self.frame_times[num][-1] + 0.5/fps)

As you can see, I have been saving time around trigger times to try to understand when/where frames can be missed. Is there a way, maybe in the callback function, to identify when exposure was done but no frame delivered? Again, we have e.g. 20050 exposure time pulses but only 20000 frames in the .avi file. We'd like to do a 1-to-1 mapping of the frames.

Let me know if you need clarifications! Thanks for your help.

Axel

[abisi]

Ah, somehow it's a bit hard to format code chunks in here. I'll try to fix that.

[TIS-Stefan]

Hello

I think, the issue is caused by the videowrite. You do not know, what happens, if frames come in, but the videowriter can not write them into the video file, because Windows is busy with something else and blocking the hard disc. This happens, when Windows flushes the disc cache. The only way I personally know of getting all images is to save them into RAM while the capture runs and save them afterwards. However, this needs a lot of RAM. The IC Express software from https://www.theimagingsource.com/en-de/support/download/icexpress-1.1.0.23/ works in that way.

A word to the issue's subject: You can not know, whether a callback fails, because it is either called and executed or not, because there is no frame. The callback is not called, while a call is currently running. Therefore, you must make sure, it's runtime is below 1/framerate seconds or your trigger frequency.

I hope, this gives a little background to you.

Stefan

[abisi]

Hello, thank you so much for your clarifications and for your help. I tried IC Express and indeed, for our needs (i.e. 2h videos, at 200 FPS with above-mentioned image format), we need hundreds of RAM per video, which makes it an infeasible. Do you know whether it's possible, with IC Express or orther, to use batches of images in RAM to be saved? For example, 5 Gb of RAM is allocated, then saved, etc.

Axel

[TIS-Stefan]

Hello Axel

Since the image creation is faster than saving single image, you have no chance with that. Thus, you must stay at saving an AVI file. I would queue the images in a list in memory. Then I would create a new thread, which saves the image from the list into the video file. But, you simply do not know, whether vid_out.write(cv2.flip(frame,0)) was successful or not. The documentation says, it writes. It does not say anything about, it does not write or errors. Thus, this approach could work, if writing into the video file is faster, than images come in. (I always wanted to try something similar, but I never had time for that.)

Stefan

[abisi]

Dear Stefan,

Thanks again for your input, this helps as I am no expert in this at all! I will try what you suggested. In particular, I will profile the callback function and check whether it's faster than the sampling period. Otherwise, I will keep looking.

Best, Axel

[TIS-Stefan]

Axel

also safe the image buffers in a FiFo (or "queue"), so images, that are not saved to the AVI file are stored in memory for later saving.

Stefan

[abisi]

Hello,

Sorry for taking so long to reply to this thread... In the end, our solution was to simply trigger each frame at the desired frame rate using IC Capture, but to save the .avi file as uncompressed (in Y800).

Thank you for your time and help!

Best, Axel

[TIS-Stefan]

Hello Axel Thank you for your feedback.