dorodnic
commented
5 years ago Hi @Liuyu010 Frame is not exactly equivalent to cv::Mat, as it carries a lot of camera specific information (link to the sensor, metadata, link to calibration, etc...) That's why there is no easy cv-->frame API. It is possible, of course. I made a modified version of im-show example demonstrating how to go about it:
// License: Apache 2.0. See LICENSE file in root directory.
// Copyright(c) 2017 Intel Corporation. All Rights Reserved.
#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <opencv2/opencv.hpp> // Include OpenCV API
int main(int argc, char * argv[]) try
{
// Declare depth colorizer for pretty visualization of depth data
rs2::colorizer color_map;
// Declare RealSense pipeline, encapsulating the actual device and sensors
rs2::pipeline pipe;
// Start streaming with default recommended configuration
pipe.start();
using namespace cv;
const auto window_name = "Display Image";
namedWindow(window_name, WINDOW_AUTOSIZE);
// Create custom processing block and its output queue:
rs2::frame_queue q;
rs2::processing_block pb(
[](rs2::frame f, rs2::frame_source& src)
{
// For each input frame f, do:
const int w = f.as<rs2::video_frame>().get_width();
const int h = f.as<rs2::video_frame>().get_height();
// frame --> cv
Mat image(Size(w, h), CV_16UC1, (void*)f.get_data(), Mat::AUTO_STEP);
// do some (silly) processing
cv::GaussianBlur(image, image, cv::Size(0, 0), 3);
// Allocate new frame. Copy all missing data from f.
// This assumes the output is same resolution and format
// if not true, need to specify parameters explicitly
auto res = src.allocate_video_frame(f.get_profile(), f);
// copy from cv --> frame
memcpy((void*)res.get_data(), image.data, w * h * 2);
// Send the resulting frame to the output queue
src.frame_ready(res);
});
pb.start(q); // Bind output of the processing block to be enqueued into the queue
while (waitKey(1) < 0 && cvGetWindowHandle(window_name))
{
rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera
rs2::frame depth = data.get_depth_frame();
// Send depth frame for processing
pb.invoke(depth);
// Wait for results
depth = q.wait_for_frame();
// Continue with the new frame
depth = depth.apply_filter(color_map);
// Query frame size (width and height)
const int w = depth.as<rs2::video_frame>().get_width();
const int h = depth.as<rs2::video_frame>().get_height();
// Create OpenCV matrix of size (w,h) from the colorized depth data
Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);
// Update the window with new data
imshow(window_name, image);
}
return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{
std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;
return EXIT_FAILURE;
}
catch (const std::exception& e)
{
std::cerr << e.what() << std::endl;
return EXIT_FAILURE;
}
Liuyu010
RealSense-Customer-Engineering
Issue Description
Hello,
I have already seen this question in the realsense community but didn't get the answer I want.
In rs_align, I have converted the rs2:: frame to cv:: Mat for using the joint bilateral filter provided by OpenCV to improve the quality of the depth map. The depth map of the processed mat type that is obtained cannot be used to call _remove_background(rs2::video_frame& other, const rs::frame & depth_frame, float depth_scale, float clippingdist).
So, I just want to know how to convert cv:: Mat to rs2:: frame. If you can help me, I would be very grateful.
Thank you!