Processing data streams asynchronously using queues

[Sweet-orange-gethub]

CameraModel: realsense D435 Operating System:linux

include <librealsense2/rs.hpp>

include <opencv2/opencv.hpp>

include

include

include

include

include "cv-helpers.hpp"

using namespace cv; using namespace rs2; using namespace std;

frame_queue f_queue;

void *a(){ pipeline pipe; pipe.start();

for (int i = 0; i < 10; i++)
    pipe.wait_for_frames();
while (1)
{
    frameset data = pipe.wait_for_frames();
    frame depth = data.get_depth_frame();
    f_queue.enqueue(depth);
    char c = waitKey(100);
    if (c == 27)
    { // ESC
        break;
    }
}

}

void *b(){ const auto window_name = "Display Image"; namedWindow(window_name, WINDOW_AUTOSIZE); colorizer colorize; while(1){ sleep(2); frame f; if(f_queue.poll_for_frame(&f)){ colorize.set_option(RS2_OPTION_COLOR_SCHEME, 2); frame ff = f.apply_filter(colorize); auto image = frame_to_mat(ff); imshow(window_name, image); } }

}

int main(int argc, char *argv[]) try {

cout << "begin" << endl;

thread tt([&] { a(); });
thread ttt([&] { b(); });
tt.join();
ttt.join();
while(1){
    cout << "main" << endl;
     sleep(1);
}

} catch (const rs2::error &e) { std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl; return EXIT_FAILURE; } catch (const std::exception &e) { std::cerr << e.what() << std::endl; return EXIT_FAILURE; } ` In fact, this code can not realize asynchronous processing of data retention. Obviously, many frames of pictures will be lost. I don't know what's wrong. I need help. Thank you sincerely.

[MartyG-RealSense]

Hi @Sweet-orange-gethub I note that you are using wait_for_frames(), which is a synchronous mode of operation rather than asynchronous. More information about this can be found at the link below.

https://github.com/IntelRealSense/librealsense/issues/3817#issuecomment-485663496

The advice in the above link suggests the SDK's rs-callback C++ sample program as an example of asynchronous operation.

https://github.com/IntelRealSense/librealsense/tree/master/examples/callback

[Sweet-orange-gethub]

Hi MartyG-RealSense Thank you for your answer,I read your answer carefully, but I didn't try.I want to reconfirm whether my description of the problem is accurate. Suppose we set the camera to receive 15 frames of images per second, but my machine can only process 5 frames per second. I don't want to miss each frame of images, so I save all the frames received by the camera in one thread and take out the frames from the queue in another thread for processing. Similar to a producer consumer model. But the problem now is that the above program can't make me pass every frame well. I can detect the existence of frame loss. When I move the camera, the image will change immediately. In theory, the image should take a long time to change. Thank you again for your reply.

[MartyG-RealSense]

If your target is 5 frames per second because that is the limit of what your machine can process then it may be simpler to use a pipeline based on wait_for_frames() and set the stream FPS at 6 instead of 15, as 6 FPS is supported for the D435 camera model for depth, color and infrared (the D455 model supports 5 FPS).

Alternatively, you can obtain a custom frame rate that is not officially supported by the camera by setting a high FPS such as 30 but only reading every 'nth' frame. For example, if FPS is set to 30 then using only every 6th frame out of 30 gives an FPS of 5. I do not have a C++ example of such a script, but the Python script in the link below that does so may give some useful insights if you wish to use that method.

https://github.com/IntelRealSense/librealsense/issues/3169#issue-404474462

[Sweet-orange-gethub]

In fact, I want to get more images (15fps), because the cameras in our project are not always working, so I can store the images obtained by the cameras in the queue first, and my machine will process them slowly (5fps). Until the queue is empty. Because the camera is not always working, my queue will not become very large. We are tracking an object. Using 5fps will degrade our accuracy, so we need a higher frame rate. However, due to the limitations of our machine, I want to use a buffer queue

[MartyG-RealSense]

A further possibility that could be considered is using the SDK's Keep() instruction to store frames in the computer's memory for future use instead of writing them to storage, and then performing batch actions such as post-processing, align and save-to-file on all of the collected frames at the same time after the pipeline is closed.

https://github.com/IntelRealSense/librealsense/issues/1942

[Sweet-orange-gethub]

The problem has been solved. Thank you again

[MartyG-RealSense]

Excellent news that you acheved a soluton, @Sweet-orange-gethub - thanks for the update!