Discrepancy between FPS and camera properties

[ilya71ru]

I have camera DMK 33UX250 I'm using example 3 (https://github.com/TheImagingSource/tiscamera/blob/master/examples/c/03-live-stream.c) I changed a line in a line of code - GstElement* pipeline = gst_parse_launch("tcambin name=source ! videoconvert ! ximagesink", &err); to "tcambin name=source ! video/x-raw,format=GRAY8,width=2448,height=2048,framerate=75/1 ! videoconvert ! appsink name=sink" FPS increased, but only to 20-25 What is the problem?

I also change some properties of the camera, such as exposure and gain, and then read what value at the time of frame acquisition. When the next frame comes, it is written that the value is already the same as I set, but in reality it only works after a few frames. How can this be fixed? Examples of changing and reading properties were taken from the example code 2 and 3, respectively.

[TIS-Stefan]

You reveice tons of frame drops. That can be caused by

• USB cable
• bad USB controller, AMD/ASMEDIA
• using front side USB connectors of your desktop pc
• bad connection of USB controller to memory.
• Weak computer, e.g. PI.

Which computer model / brand do you use? How do you measure the fps?

[ilya71ru]

I use Khadas Vim3, but it is not loaded even by a quarter at the same time I measure fps by counting callback calls per second, or I look at the metadata from example 10

[TIS-Stefan]

I have totally no experience with your computer. But the usual case is a the USB controller not being able to save incoming image data in time into memory. You may try

tcambin name=source ! video/x-raw,format=GRAY8,width=2448,height=2048,framerate=75/1 ! appsink name=sink

The video convert is not needed, because the appsink will accept the GRAY 8. Also I would lower the frame rate in order to check, where the limit is, that can be handled. You can achieve higher frame rates, if you use e.g. 30/1 or 40/1, instead of maximum, because the data packages are send slower by the camera.

I am very sorry, but I would not expect to achieve the full frame rate on your computer.

Stefan

[ilya71ru]

video converter does not affect fps in my case. When I set fps = 30, I get 10, when 5, I get 3.5

But this is not the main problem, I can work on such fps.

Could you suggest a solution to the second problem "I also change some properties of the camera, such as exposure and gain, and then read what value at the time of frame acquisition. When the next frame comes, it is written that the value is already the same as I set, but in reality it only works after a few frames. How can this be fixed? Examples of changing and reading properties were taken from the example code 2 and 3, respectively."

[ilya71ru]

Maybe I could get exposure and gain through metadata? Perhaps this will solve the problem? But I don't see in the metadata example how to change it to achieve this.

[TIS-Stefan]

I am sorry, but the frame meta data do not contain these properties. If the camera is not in trigger mode, the exposure time and gain become effective after the next or 2nd next frame. While the sensor is exposing, it can not set registers. That is delayed, until a frame has been completed. Therefore, you need to wait for one or two frames.

Stefan

[ilya71ru]

Okay, I get it, thanks. Could you also tell me with the work on the trigger. I repeated example 6 by adding the callback from example 3 to it. But the callback doesn't work, what did I do wrong?

[TIS-Stefan]

I am sorry, but I need to see you code, in order to see, what is wrong. What do you mean by "callback doesn't work"? Is it not called?

Stefan

[ilya71ru]

bool TisCamera::init()
    {
        gst_debug_set_default_threshold(GST_LEVEL_WARNING);
        gst_init(nullptr, nullptr);

        const char* serial = nullptr; // the serial number of the camera we want to use
        const char* pipeline_str = "tcambin name=source ! video/x-raw,format=GRAY8,width=2448,height=2048,framerate=75/1 ! appsink name=sink";

        GError* err = nullptr;
        _pipeline = gst_parse_launch(pipeline_str, &err);

        /* test for error */
        if (!_pipeline)
        {
            L_ERROR("Could not create pipeline. Cause: %s\n", err->message);
            return false;
        }

        _source = gst_bin_get_by_name(GST_BIN(_pipeline), "source");

        if (serial)
        {
            GValue val = {};
            g_value_init(&val, G_TYPE_STRING);
            g_value_set_static_string(&val, serial);

            g_object_set_property(G_OBJECT(_source), "serial", &val);
            gst_object_unref(_source);
        }

        /* retrieve the appsink from the pipeline */
        GstElement* sink = gst_bin_get_by_name(GST_BIN(_pipeline), "sink");

        // tell appsink to notify us when it receives an image
        g_object_set(G_OBJECT(sink), "emit-signals", TRUE, nullptr);

        // tell appsink what function to call when it notifies us
        g_signal_connect(sink, "new-sample", G_CALLBACK(callback), this);

        GValue trigger = {};
        g_value_init(&trigger, G_TYPE_BOOLEAN);
        g_value_set_boolean(&trigger, TRUE);

        tcam_prop_set_tcam_property(TCAM_PROP(_source), "Trigger Mode", &trigger);

        gst_element_set_state(_pipeline, GST_STATE_PLAYING);

        GValue trigger_val = {};
        g_value_init(&trigger_val, G_TYPE_BOOLEAN);
        g_value_set_boolean(&trigger_val, TRUE);

        gboolean r = tcam_prop_set_tcam_property(TCAM_PROP(_source), "Software Trigger", &trigger_val);
        if (!r)
        {
            printf("!!! Could not trigger. !!!\n");
        }
        else
        {
            printf("=== Triggered image. ===\n");
            //sleep(2);
            //callback(sink, this);
        }

        gst_object_unref(sink);

        return true;
    }

static GstFlowReturn callback(GstElement* sink, void* user_data);
GstFlowReturn TisCamera::callback(GstElement* sink, void* user_data)
    {
        TisCamera* _this = reinterpret_cast<TisCamera*>(user_data);

        GstSample* sample = nullptr;
        /* Retrieve the buffer */
        g_signal_emit_by_name(sink, "pull-sample", &sample, nullptr);

        if (sample)
        {
            GstBuffer* buffer = gst_sample_get_buffer(sample);
            GstMapInfo info; // contains the actual image

            GstMeta* meta = gst_buffer_get_meta(buffer, g_type_from_name("TcamStatisticsMetaApi"));

            if (meta)
            {
                printf("We have meta\n");
            }
            else
            {
                g_warning("No meta data available\n");
            }

            GstStructure* struc = ((TcamStatisticsMeta*)meta)->structure;

            // this prints all contained fields
            gst_structure_foreach(struc, meta_struc_print, NULL);

            if (gst_buffer_map(buffer, &info, GST_MAP_READ))
            {
                GstVideoInfo* video_info = gst_video_info_new();
                if (!gst_video_info_from_caps(video_info, gst_sample_get_caps(sample)))
                {
                    // Could not parse video info (should not happen)
                    L_WARN("Failed to parse video info");
                    return GST_FLOW_ERROR;
                }

                /* Get a pointer to the image data */
                unsigned char* data = info.data;

                /* Get the pixel value of the center pixel */
                const auto stride = video_info->stride[0];

                gst_buffer_unmap(buffer, &info);
                gst_video_info_free(video_info);
            }

            // delete our reference so that gstreamer can handle the sample
            gst_sample_unref(sample);
        }
        return GST_FLOW_OK;
    }

[TIS-Stefan]

The code looks fine to me. Did you set a break pointer in the callback function, to seen whether it was called? Or you may make a printf() in the callback function. Also you should wait a moment instead of ending the program immediately after you made the software trigger. The image needs a few milliseconds (1/framerate) until it is in the computer and then it may is processed a little bit and then the callback is called.

Stefan

[ilya71ru]

my program does not terminate and works for a long time, but the callback function is not called. In this function I was doing a printf or a breakpoint and I don't see it working

[ilya71ru]

I tried to call the callback function myself, but then it hangs on the line g_signal_emit_by_name(sink, "poll-sample", &sample, nullptr)

Maybe you have an example a little more complicated than 06-software trigger.c where I could see how you get an image by trigger in c++

[TIS-Stefan]

Is the callback called, if the trigger mode is disabled?

tried to call the callback function myself, but then it hangs on the line g_signal_emit_by_name(sink, "poll-sample", &sample, nullptr) That will not be that helpful.

[ilya71ru]

Вызывается ли обратный вызов, если режим триггера отключен?

сам пытался вызвать функцию обратного вызова, но затем она зависает в строке g_signal_emit_by_name (раковина, "poll-sample", & sample, nullptr). Это не поможет.

Yes

[TIS-Stefan]

The the call to the software trigger does not work as expected. Please pass an integer instead of a true to the software trigger.

[ilya71ru]

        GValue trigger_val = {};
        g_value_init(&trigger_val, G_TYPE_INT);
        g_value_set_int(&trigger_val, 1);

        gboolean r = tcam_prop_set_tcam_property(TCAM_PROP(_source), "Software Trigger", &trigger_val);

then I get an error: 0:00:02.330112544 164039 0x555555b27c00 ERROR   tcammainsrc gsttcammainsrc.cpp:1001:gst_tcam_mainsrc_set_caps: Unable to set format in device 0:00:02.330159745 164039 0x555555b27c00 WARN   basesrc gstbasesrc.c:3072:gst_base_src_loop: error: Internal data stream error. 0:00:02.330171678 164039 0x555555b27c00 WARN   basesrc gstbasesrc.c:3072:gst_base_src_loop: error: streaming stopped, reason not-negotiated (-4)

I was able to trigger the trigger hardware and then everything works fine.

[ilya71ru]

I was able to raise FPS by removing some of my functions from the callback function. Then the FPS reaches 30. I switched from a single-board computer to a regular linux laptop and checked using cam-capture and there I get 75 FPS. But my callback cannot achieve this without any of my functions

[TIS-Stefan]

I tested the "06-softwaretrigger.c" sample with an 33U camera.

GValue trigger_val = {};
g_value_init(&trigger_val, G_TYPE_BOOLEAN);
g_value_set_boolean(&trigger_val, TRUE);
boolean r = tcam_prop_set_tcam_property(TCAM_PROP(source), "Software Trigger", &trigger_val);

Works perfectly on my Ubuntu 20.04. I extended the sample by a callback:

#include <unistd.h> /* sleep  */
#include <stdio.h>  /* printf */
#include <gst/gst.h>
#include <tcamprop.h>

static GstFlowReturn callback(GstElement* sink, void* user_data)
{
    GstSample* sample = NULL;
    /* Retrieve the buffer */
    g_signal_emit_by_name(sink, "pull-sample", &sample, NULL);
    if (sample)
    {
        printf("Callback!\n");
         gst_sample_unref(sample);
    }
    return GST_FLOW_OK;
}

int main (int argc, char *argv[])
{
    /* this line sets the gstreamer default logging level
       it can be removed in normal applications
       gstreamer logging can contain verry useful information
       when debugging your application
       # see https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html
       for further details
    */
    gst_debug_set_default_threshold(GST_LEVEL_WARNING);

    gst_init(&argc, &argv); // init gstreamer

    const char* serial = NULL ; // the serial number of the camera we want to use

    GError* err = NULL;

    GstElement* pipeline = gst_parse_launch("tcambin name=source ! appsink name=sink", &err);

    /* test for error */
    if (pipeline == NULL)
    {
        printf("Could not create pipeline. Cause: %s\n", err->message);
        return 1;
    }

    GstElement* source = gst_bin_get_by_name(GST_BIN(pipeline), "source");

    if (serial != NULL)
    {
        GValue val = {};
        g_value_init(&val, G_TYPE_STRING);
        g_value_set_static_string(&val, serial);

        g_object_set_property(G_OBJECT(source), "serial", &val);

        g_value_unset(&val);
    }

    /* retrieve the appsink from the pipeline */
    GstElement* sink = gst_bin_get_by_name(GST_BIN(pipeline), "sink");

    // tell appsink to notify us when it receives an image
    g_object_set(G_OBJECT(sink), "emit-signals", TRUE, NULL);

    // tell appsink what function to call when it notifies us
    g_signal_connect(sink, "new-sample", G_CALLBACK(callback), NULL);

    gst_element_set_state(pipeline, GST_STATE_PLAYING);

    /*
      This sleep exists only to ensure
      that a live image exists before trigger mode is activated.
      for all other purposes this can be removed.
     */
    sleep(2);

    GValue trigger_val = {};
    g_value_init(&trigger_val, G_TYPE_BOOLEAN);
    g_value_set_boolean(&trigger_val, TRUE);

    // Check for the type of trigger
    // Depending on the camera model trigger may be a bool or an enum
    const gchar* trigger_mode_type =  tcam_prop_get_tcam_property_type(TCAM_PROP(source),
                                                                       "Trigger Mode");

    if (!trigger_mode_type)
    {
        printf("Unable to determine type of 'Trigger Mode'\n");
        exit(1);
    }

    gboolean trigger_is_bool = TRUE;
    GValue trigger = {};

    if (strcmp(trigger_mode_type, "enum") == 0)
    {
        trigger_is_bool = FALSE;
        g_value_init(&trigger, G_TYPE_STRING);
        g_value_set_string(&trigger, "On");
        tcam_prop_set_tcam_property(TCAM_PROP(source), "Trigger Mode", &trigger);
    }
    else
    {
        g_value_init(&trigger, G_TYPE_BOOLEAN);
        g_value_set_boolean(&trigger, TRUE);

        tcam_prop_set_tcam_property(TCAM_PROP(source), "Trigger Mode", &trigger);
    }

    while (0 == 0)
    {
        printf("Press 'q' then 'enter' to stop the stream.\n");
        printf("Press 'Enter' to trigger a new image.\n");

        char c = getchar();

        if (c == 'q')
        {
            break;
        }

        gboolean r = tcam_prop_set_tcam_property(TCAM_PROP(source), "Software Trigger", &trigger_val);
        if (!r)
        {
            printf("!!! Could not trigger. !!!\n");
        }
        else
        {
            printf("=== Triggered image. ===\n");
        }
    }

    /* deactivate trigger mode */
    /* this is simply to prevent confusion when the camera ist started without wanting to trigger */

    if (trigger_is_bool)
    {
        g_value_set_boolean(&trigger, FALSE);
        tcam_prop_set_tcam_property(TCAM_PROP(source), "Trigger Mode", &trigger);
    }
    else
    {
        g_value_set_string(&trigger, "Off");
        tcam_prop_set_tcam_property(TCAM_PROP(source), "Trigger Mode", &trigger);
    }

    g_value_unset(&trigger);
    g_value_unset(&trigger_val);

    // this stops the pipeline and frees all resources
    gst_element_set_state(pipeline, GST_STATE_NULL);

    gst_object_unref(source);
    /* the pipeline automatically handles all elements that have been added to it.
       thus they do not have to be cleaned up manually */
    gst_object_unref(pipeline);

    return 0;
}

You may try that sample. You will see, that the callback is called after pipeline set to PLAYING state many times, until the camera is set into trigger mode. Then you can hit the Enter key to make software triggers.

Stefan

[TIS-Stefan]

FPS Question:

I was able to raise FPS by removing some of my functions from the callback function. Then the FPS reaches 30. I switched from a single-board computer to a regular linux laptop and checked using cam-capture and there I get 75 FPS. But my callback cannot achieve this without any of my functions

If your single board computer is to weak, then a high CPU load can lead to incomplete image data transfer. Therefore, I suggest to use lower frame rates on the single board computer.

Stefan

[ilya71ru]

Вопрос FPS:

Мне удалось поднять FPS, удалив некоторые из моих функций из функции обратного вызова. Затем FPS достигает 30. Я переключился с одноплатного компьютера на обычный ноутбук с Linux, проверил с помощью cam-capture и получил 75 FPS. Но мой обратный вызов не может достичь этого без каких-либо моих функций

Если ваш одноплатный компьютер слишком слабый, высокая загрузка ЦП может привести к неполной передаче данных изображения. Поэтому я предлагаю использовать более низкую частоту кадров на одноплатном компьютере.

Стефан

the last described actions I did on a laptop with a core i7 processor

[ilya71ru]

I tried your code. The software trigger is working. But the maximum FPS on my laptop still does not exceed 30.

[TIS-Stefan]

But the maximum FPS on my laptop still does not exceed 30. Does this mean, only 30 fps max are available, not 75? If yes, then the camera may is connected to an USB 2.0 port. You can send the output of lsusb -t It seems to be a computer issue. Stefan

[ilya71ru]

The problem is definitely not in the computer, because when I run app team-capture written in python, I get 75 FPS

[TIS-Stefan]

The maximum trigger frequency you can use is determined by exposuretime + 1/framerate. You must use a low exposure time, if you want to trigger the camera with 75Hz at 75ps in the camera. I suggest to start with 60Hz and very low exposure time and check how much you can rise the trigger frequency. In my tests exposure times less than 1/300 second do the job.

[TIS-Edgar]

Closed due to inactivity.