search

princeton-vl / RAFT-Stereo

MIT License
673 stars 129 forks source link

Question about generating pointcloud #26

Open Ridhwanluthra opened 2 years ago

Ridhwanluthra commented 2 years ago

While generating pointcloud from the disparity generated using raft-stereo I end up getting something like thi: raft_ptc ) Where there is a trail of points from the object to the background. The image I have posted is a top view. I am using the standard equations to generate the pointcloud, am I missing something?

lahavlipson commented 2 years ago

I observed this as well, especially near object boundaries which end up getting smoothed by the network.

The results look a lot cleaner if you filter the points using forward-backward consistency. The bilinear_sampler function should make it easy to sample a tensor using the predicted coordinates

Ridhwanluthra commented 2 years ago

Oh interesting, thank you so much, I'll look into it. Additionally, do you think their could be a way to get the network to not do this?

lahavlipson commented 2 years ago

Any network that uses an endpoint error loss for training will probably smooth over some object boundaries, so I'm not sure.

Also the point cloud may look better (but sparser) if you remove points whose gradient is larger than some threshold.

Mythili-kannan commented 2 years ago

I am also facing same issue(while generating Point clouds getting trailing points), can you please explain little more on how bilinear_sampler can be used to filter the trailing points and also how to calculate gradient of points in pointcloud to remove, it will be very helpful.

lahavlipson commented 2 years ago

You can use forward-backward consistency to filter the points. The bilinear_sampler() function in this repo is just a wrapper for torch.nn.functional.grid_sample, which will enable you to sample the backward flowfield using a forward one. You can produce a "backward" flowfield by just reversing the images and rerunning RAFT-Stereo.

A simpler method is to just exclude points whose predicted depth is very different from that of its neighbors.

lahavlipson commented 2 years ago

On the Readme I just put an example of a point cloud filtered using the second approach.

lahavlipson commented 1 year ago

In case anyone comes across this, here is a Google Colab with code to produce a point cloud from the RAFT-Stereo predictions: https://colab.research.google.com/drive/1G8WJCQt9y55qxQH6QV6PpPvWEbd393g2?usp=sharing

niubiplus2 commented 3 weeks ago

Hello, how to get the baseline of the cameras, please?