search

sniklaus / softmax-splatting

an implementation of softmax splatting for differentiable forward warping using PyTorch
468 stars 58 forks source link

about bilinear kernel #44

Closed 863689877 closed 3 years ago

863689877 commented 3 years ago

Thank you for your nice work! The bilinear kernel in your paper is descriped as followed: As far as I know, bilinear operation is used on four regular two-dimensional points, and how to used it on the an irregular and arbitrary number of warped points? Please explain this problem, thanks.

sniklaus commented 3 years ago

I am afraid that I do not understand your question, my apologies. The bilinear kernel essentially means that we splat a pixel to four pixels, where the contribution to each of the four pixels is determined by the bilinear kernel.

https://github.com/sniklaus/softmax-splatting/blob/7ba7a19dc054674ae0407674372b6f0d8a36b2c7/softsplat.py#L32-L51

863689877 commented 3 years ago

A warped pixel value is equal to the following equation by deducing the bilinear interpolation formula. valueInput = fltNorthwest * valueNorthwest + fltNortheast * valueNortheast + fltSouthwest * valueSouthwest + fltSoutheast * valueSoutheast According to your interpretation, the values of the four top angle pixels are calculated by the following equation: valueNorthwest = valueInput_1 * fltNorthwest_1 + valueInput_2 * fltNorthwest_2 + ... + valueInput_k * fltNorthwest_k In other words: valueNorthwest = valueInput * fltNorthwest I don`t think it is reasonable.

sniklaus commented 3 years ago

My apologies, but I don't understand what you are saying.

863689877 commented 3 years ago

微信图片_20211021111240

sniklaus commented 3 years ago

That seems to match my implementation?

863689877 commented 3 years ago

It matches the equation as followed: valueInput = fltNorthwest * valueNorthwest + fltNortheast * valueNortheast + fltSouthwest * valueSouthwest + fltSoutheast * valueSoutheast But it does not match the equation which is implemented in your paper and code: valueNorthwest = valueInput_1 * fltNorthwest_1 + valueInput_2 * fltNorthwest_2 + ... + valueInput_k * fltNorthwest_k valueNorthwest = valueInput * fltNorthwest Please notice the difference between the two equations.

sniklaus commented 3 years ago

Sorry, I don't get it. But since the code is available, why don't you just change it to what you think is correct/better and give it a try?

863689877 commented 3 years ago

I do not think there is something wrong with your code, I do not understand how splatting operation works, because I can only derive interpolation formula, but I do not know how to derive splatting formula from interpolation formula. Please clarify the difference between interpolation and splatting operations.

sniklaus commented 3 years ago

Think of it as in inverse of the bilinear interpolation. Instead of taking a weighted sum of four pixels to compute the interpolated pixel, in splatting we have to distribute the pixel that we are splatting to the four closest pixels in the target. The weights that are used during splatting (scattering) happen to be the same as the weights used during interpolation (gathering).

863689877 commented 3 years ago

Thank you for your patient reply! I found the key of the problem. How splatting weights are obtained and why they happen to be the same as interpolation weights. Is there a theoretical derivation or is it an intuitive assumption? Could you please give the derivation process or relevant literature that can provide the answer.