search

sniklaus / softmax-splatting

an implementation of softmax splatting for differentiable forward warping using PyTorch
456 stars 60 forks source link

splatting for 3d voxels #29

Closed kwea123 closed 3 years ago

kwea123 commented 3 years ago

In the neural scene flow fields paper you use 3d splatting to compute time interpolation views. I wonder how exactly it is done, do you splat the 3d points themselves, or do you project them onto the image plane first then splat the 2d pixels?

It seems that the current implementation only supports 4D tensors (i.e. 2d images splatting).

sniklaus commented 3 years ago

Thank you for your interest in our work! The splatting in "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes" is done in image space. While the scene flow is in 3D, the scene flow in image space is 2D as usual.

kwea123 commented 3 years ago

Then what if the object's motion is mainly towards (or away from) the camera? If you predefine a fixed number of planes (as what I understand from figure 2 in the supplementary), the flow might splat points towards a plane that doesn't exist. Unless you place a lot of planes, if you only project them to the nearest plane, I think the performance will degrade. What do you think?

sniklaus commented 3 years ago

The splatting is done in 2D image space, I am afraid that there are no planes involved. You just have to determine how a pixel moves in image space given its scene flow and then splat it, but do so back to front (starting with the pixels/points far away). I hope that the code for "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes" will be released soon, which may help you with your efforts.

kwea123 commented 3 years ago

The code is out!

Indeed it's like what I thought: he uses multiple "planes" (it's sample points actually, but since all sample points on the rays lie on the same planes, so I refer this as planes), splats each one and uses traditional MPI to render. https://github.com/zhengqili/Neural-Scene-Flow-Fields/blob/4f48bfc3872bfb735850e8bb969a0bc1d56de234/nsff_exp/render_utils.py#L157-L163 Thanks again for your answer, I will post issue on his repo if I have further questions.

sniklaus commented 3 years ago

Sorry about the confusion, I can see now why you consider them to be planes. It is just part of NeRF though, moving along the rays at discrete steps. We just do it in a wavefront-manner, instead of ray by ray.