Reprojection yielded 0 valid points :(

[eric-schleicher]

Greetings @sdmiller

Tying to get up and running with cpu_tsdf (thanks for making it!).

I'm exporting data from wonderful RTABMAP in an attempt to run it throught cpu_tsdf because such things are exciting to me. Everything looks like it compiled just fine, but i'm getting a error i don't quite understand.

I've created the pose files, put them in the right place (alongside the pcds)

And formatted the content as indicated in the instructions.

When i run the integrate command it runs very quicky. with --verbose i can see that the pose files are parsed correctly, but it seems the issue is that Reprojection yielded 0 valid points, of initial 28644 on each of the pcd frames that I've provided.

Any tips on what the issue might be? is it that the size of the volume is incorrect? or not matched to the unit system of the poses?

Reprojection yielded 0 valid points, of initial 28395
Cloud bounds: [inf, -inf], [inf, -inf], [inf, -inf]
On frame 40 / 430

Should it be producing some bounds (not inf)? might it be an issue with the pcd files? it does appear that the correct number of points are being found.

Any help/pointers here would be greatly appreciated.
The subect material is cool too (a russian submarine) as you can see (click to see on sketchfab) .

I would love to see how much better the tsdf will be than this one (which was created with poisson).

verbose output from the command here.

https://gist.github.com/eric-schleicher/2cf75842a597a41d953e8312459c4b16

[sdmiller]

Sorry for the delay! Been traveling.

Yes, so it looks to me like this is failing during the "Reprojection" phase. That's something I added strictly to the integrate script, since a bunch of people were working with unorganized clouds.

Basically, the TSDF algorithm assumes it is given a series of organized clouds (such that cloud(u,v) corresponds to pixel (u,v)). Since most average users were not happening to pass in clouds of this variety, I added a little trick to the script which synthetically re-roganized the cloud. It does this using the intrinsics of the sensor (focal length, center pixel) to backproject the points into pixel space. By default we use the intrinsics of a standard kinect sensor.

That leads to a few things:

• If your data is already organized, you can pass in the "--organized" flag to tell the script not to do this reprojection. BUT
• Regardless of whether it is organized, if reprojection is completely failing here it probably means the camera intrinsics (--fx,-- fy, --cx, --cy, --width, --height) are drastically different from what we expect them to be. You'll want those to be correct to get reasonable results from the TSDF.
• I'm still shocked that reprojection worked in literally 0 cases. This indicates to me that the clouds are extremely different from what we are expecting. I'd love to take a look at one and see.

When in doubt, I'd suggest hopping into integrate.cpp and looking at the for loop where it reprojects points. A printout or two there could help dramatically.

[sdmiller]

Also worth noting: if your data is organized and you don't know the intrinsics, you can use my handy little get_intrinsics tool (src/prog/get_intrinsics.cpp) to get a reasonably good estimate

[eric-schleicher]

Thanks for taking the time to response and explain why I might be seeing this. I shared the issue with @matlabbe the maintainer of RTAB_MAP and he implemented cpu_tsdf as a post processing step. he clearly understood the detail needed to integrate the libary directly into the application.

I think we can close, since the issue has been addressed and there is a working solution now... but if you want to see the organized pointcloud that rtabmap produces, can certainly share a dataset with you.

let me know, and thanks for following through!

[matlabbe]

The problem was that RTAB-Map exported organized clouds, but in a different frame (e.g., map frame). CPU-TSDF seems to assume that organized clouds are in camera frame (which makes sense). I added new output frame option to RTAB-Map when exporting not assembled organized clouds so that they can be exported in camera frame (z forward, x right, y down). The Export poses action has been also updated to select in which frame we want the poses (here in camera frame). So if there are similar or future implementations similar to cpu-tsdf, we could get the data right!

cheers, Mathieu

[eric-schleicher]

Thanks so much @matlabbe!