search

foxtierney / RealSense2OpenPose3D

An open source program for generating 3D keypoint files with an Intel RealSense depth camera and OpenPose
BSD 3-Clause "New" or "Revised" License
16 stars 2 forks source link

Framerate Dramatically Limited by Frame Alignment and Point Cloud Calculation #1

Open foxtierney opened 1 year ago

foxtierney commented 1 year ago

In the main loop of RealSense2OpenPose3D.cpp, the lines...

fsAligned = align.process(fsAligned); //Align the depth to the color frame
rs2::frame depthAligned = fsAligned.get_depth_frame(); //Get the aligned depth frame

//Generate a point cloud and a vertex array from that to hand to "updateKeypoints()" so that it can find the world coordinates
rs2::pointcloud ptCld = rs2::pointcloud();
rs2::points depthPoints = ptCld.calculate(depthAligned);
depthVertices = depthPoints.get_vertices();

...account for about 45-60ms of delay when using the chrono library to do some speed testing. This limits the framerate to 12fps to be extra safe. If OpenPose were to run faster than RS2OP3D, then there is the possibility of losing real-time-ness.

I cannot see how to multi-thread this process to speed it up. Perhaps there is another way that these depth points in 3D space can be acquired instead of with this slow vector. Or maybe there are improvements that can be made to the SDK to reduce latency.

Any help on this would be appreciated.

foxtierney commented 3 months ago

After talking to a co-worker about this problem, it seems there might be a solution related to the rs2_deproject_pixel_to_point() function. "Documentation" page: https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20 Function definition: https://github.com/IntelRealSense/librealsense/blob/5e73f7bb906a3cbec8ae43e888f182cc56c18692/include/librealsense2/rsutil.h#L46

When I have time in the coming month or two, I should be able to get this working. Otherwise, I hope someone else takes a stab at this!