search

MIT-SPARK / Kimera-Semantics

Real-Time 3D Semantic Reconstruction from 2D data
BSD 2-Clause "Simplified" License
633 stars 134 forks source link

Accept camera streams with different frame id's #25

Closed violetteavi closed 4 years ago

violetteavi commented 4 years ago

I'm making the metric reconstruction portion of Kimera-Semantics work with the Intel RealSense d435i. I've figured out how to feed the depth and left IR stream into Kimera-Semantics (see this PR), but receive this error.

[ERROR] [1586842960.894309340]: Depth image frame id [camera_depth_optical_frame] doesn't match RGB image frame id [camera_infra1_optical_frame]

It seems that there is a check to make sure the depth and rgb stream are aligned by ensuring they are on the same frame. According to tf_get, camera_depth_optical_frame and camera_infra1_optical_frame have a transform of zero-- they are on the same link in all but name.

Can we change the check such that it ensures the frames have zero relative translation/rotation relative to each other, instead of just checking that they are on the same frame?

violetteavi commented 4 years ago

It looks like it's not a problem with the Kimera-Semantics, but with the PointCloudXyzrgbNodelet and hardware registration? Unsure, see this thread

violetteavi commented 4 years ago

I've implemented a launch file with depth_image_proc/register in the PR. This solved the frame mismatch error, but the output mesh is still empty.

@ToniRV Any ideas?

violetteavi commented 4 years ago

Using depth_image_proc/register worked once we properly added the nodelet manager! Now having some trouble recording the static transform between the depth stream and infrared stream, but that's a different issue.