ERROR tcam-libtcam timeout while waiting for new image buffer

marcoabrate commented 1 year ago

Describe the bug I am trying to record videos from a DMK 37AUX273 camera on a Raspberry Pi 4 model B, using Python. I am facing an issue with a recurring warning and a recurring error, both distruptive the recording by either shortening, corrupting or stopping the recording. The warning is:

0:05:02.151105863  3009 0xf0606180 WARN            tcam-libtcam V4l2Device.cpp:1266:stream: Did not receive image for long time.

and the error:

0:00:56.567113482  3349 0xeff06580 ERROR           tcam-libtcam V4l2Device.cpp:1244:stream: Timeout while waiting for new image buffer.

To Reproduce The warnings and errors occur multiple times with the scripts I am running. I am running three scripts to save videos as frames, AVI, and MP4 files. Here's the script that saves frames:

#!/usr/bin/env python3

import time
import sys
import gi
import os
from datetime import datetime

gi.require_version("Gst", "1.0")

from gi.repository import Gst

def main():
    Gst.init(sys.argv)

    Gst.debug_set_default_threshold(Gst.DebugLevel.WARNING)

    # Set this to a serial string for a specific camera
    serial = "31320403"

    # To save frames as separate JPG images:
    pipeline = Gst.parse_launch(
        "tcambin name=source"
        " ! video/x-raw,format=BGRx,width=1440,height=1080,framerate=15/1"
        " ! videoconvert"
        " ! jpegenc"
        " ! multifilesink name=fsink"
    )
    suffix = datetime.now().strftime("%d%m%Y_%H%M%S")
    save_dir = f"./saved_videos/saved_stream_{suffix}"
    if not os.path.exists(save_dir):
        os.makedirs(save_dir)
    save_path = os.path.join(save_dir, "frame%06d.jpg")
    fsink = pipeline.get_by_name("fsink")
    fsink.set_property("location", save_path)

    camera = pipeline.get_by_name("source")
    if serial:
        camera.set_property("serial", serial)

    pipeline.set_state(Gst.State.PLAYING)

    stream_time = 3600 # seconds

    start_time = time.time()
    print(f"[+] starting video stream for {int(stream_time/60)} minutes")
    print("[*] or press Ctrl-C to stop")
    try:
        while True:
            seconds = time.time() - start_time
            if seconds%600 == 0 : print(f"[*] recorded for {int(seconds/60)} minutes")
            if seconds > stream_time : break
    except KeyboardInterrupt:
        pass
    finally:
        print("[*] closing stream")
        # cleanup, reset state
        pipeline.set_state(Gst.State.NULL)

    return 0

if __name__ == '__main__':
    main()

Example output that corrupts the recording (i.e. the number of frames saved is much smaller than seconds*FPS):

pi@raspberrypi:~/project $ python3 save_stream_frames.py
[+] starting video stream for 5 minutes
[*] or press Ctrl-C to stop
0:00:56.567113482  3349 0xeff06580 ERROR           tcam-libtcam V4l2Device.cpp:1244:stream: Timeout while waiting for new image buffer.
0:02:38.637817130  3349 0xeff06580 ERROR           tcam-libtcam V4l2Device.cpp:1244:stream: Timeout while waiting for new image buffer.
0:02:53.149090979  3349 0xeff06580 ERROR           tcam-libtcam V4l2Device.cpp:1244:stream: Timeout while waiting for new image buffer.
0:04:33.191870729  3349 0xeff06580 ERROR           tcam-libtcam V4l2Device.cpp:1244:stream: Timeout while waiting for new image buffer.
[*] closing stream
pi@raspberrypi:~/project $

The other two scripts are mostly the same, only the pipeline changes. For AVI

pipeline = Gst.parse_launch(
        "tcambin name=source"
        " ! video/x-raw,format=BGRx,width=1440,height=1080,framerate=15/1"
        " ! videoconvert"
        " ! avimux"
        " ! filesink name=fsink"
    )

and for MP4:

pipeline = Gst.parse_launch(
        "tcambin name=source"
        " ! video/x-raw,format=BGRx,width=1440,height=1080,framerate=15/1"
        " ! videoconvert"
        " ! x264enc"
        " ! mp4mux"
        " ! filesink name=fsink"
    )

Expected behavior The scripts should save the videos of the specified length without errors.

computer used (please complete the following information):

OS: Raspbian GNU/Linux 11 (bullseye)
Architecture/Platform: Raspberry Pi 4 model B
Camera model you use: DMK 37AUX273

tiscamera version (tcam-ctrl --version, if possible):

Versions: 
Tcam:   1.1.1_master/ee45fcf7_rev_4142
Aravis: 0.8_version_e977fa4
Modules:    gst_aravis_arvu3v_v4l2_libusb_tools_doc_tests

thank you

TIS-Stefan commented 1 year ago

Hello

The error message tells us, there are no more (complete) images received from the camera (obviously). This can happen, if the CPU load is too high or the bus in the PI is full. However, 15 fps at this resolution is not too much data, so there should be no issues with USB 3 connections. I suppose, you connected the camera to the USB 3 ports of the PI 4

Example output that corrupts the recording (i.e. the number of frames saved is much smaller than seconds*FPS):

That is expected, because the SD card is not fast enough for saving so many single images per second. There is a similar issue in Windows, when Windows flushes the disc-cache, no more write operations are possible

A first step for narrowing down the issue is to use the xvimagesink instead of saving the images:

  pipeline = Gst.parse_launch(
        "tcambin name=source"
        " ! video/x-raw,format=BGRx,width=1440,height=1080,framerate=15/1"
        " ! videoconvert"
        " ! xvimagesink"
    )

If this works well, then the issue is at saving only. Which makes the things more complicated, because we have to read in the Gstreamer documentation about how to optimize the parameters for the used encoders. This is a pure software issue then. If I remember correctly, there were the omx* GStreamer modules for compression, which make a better use of the PI's hardware. You may try them.

Stefan

marcoabrate commented 1 year ago

Hi @TIS-Stefan and thank you. I understand how the problem could be at the SD card level, and we did make sure to buy the fastest SD cards available to avoid bottlenecks in the saving process. I will try this xvimagesink.

However, this problem persists when saving with the AVI and MP4 formats, do you have any ideas why this might be the case?

TIS-Kevin commented 1 year ago

Hi marcoabrate,

However, this problem persists when saving with the AVI and MP4 formats, do you have any ideas why this might be the case? Currently not.

But i would like you to be sure about the connection between Raspi 4 and camera. Is the camera connected to a USB 3 Port? Also what type of cable is used? Some Cables with a Type C Connector especially for Smartphones might not work correctly with our cameras.

Also let us know your results with the xvimagesink

marcoabrate commented 1 year ago

hi @TIS-Kevin, sorry for the late reply. The camera is connected to the Raspi 4 with the Amazon Basics usb-c to usb-3.0 cable.

running with the xvimagesink does not generate any errors and I can see the videos without any frame dropping or other issues. At this point the problem must be at saving time, I will check the omx* GStreamer modules for compression @TIS-Stefan suggested.

TIS-Stefan commented 1 year ago

Hello I guess, an SD card is too slow to save video video files with 1440x1024@15 fps. I never tried that on my own. If I would need to save video files on a PI, I would try something like a RAM disc, save the video there first and copy it later to SD card. Stefan

TheImagingSource / tiscamera

ERROR tcam-libtcam timeout while waiting for new image buffer #536