search

sniklaus / softmax-splatting

an implementation of softmax splatting for differentiable forward warping using PyTorch
468 stars 58 forks source link

Comparison with other work #37

Closed rsjjdesj closed 3 years ago

rsjjdesj commented 3 years ago

Hello, Nice work. Have you compared your work with other recent methods such as RIFE, XVFI etc., especially on high motion content.

sniklaus commented 3 years ago

Thank you for the kind words. SoftSplat was published well before RIFE and XVFI, so you will have to look into the papers of RIFE and XVFI for comparisons. You can find a comparison of RIFE and SoftSplat in Table 2 of the RIFE paper. It shows that RIFE-Large is similar to SoftSplat on Vimeo90k but is outperformed by SoftSplat on UCF101 and Middlebury.

image

Unfortunately, XVFI does not compare to SoftSplat in their paper but you can do the comparison yourself quite easily. In the XVFI repository, they quote a PSNR of 35.07 on Vimeo90k [1] whereas SoftSplat achieves 36.10 in this benchmark (a big difference).

If you want to compare SoftSplat/RIFE/XVFI on footage with large motion, you could run RIFE/XVFI on the following benchmark and compare it to the SoftSpalt results (Table 2 of our paper): https://github.com/sniklaus/softmax-splatting/blob/master/benchmark.py

[1] https://github.com/JihyongOh/XVFI#quick-start-for-vimeo90k-as-in-fig-8

rsjjdesj commented 3 years ago

Thanks for the details. Yes, I understand about the objective metrics. However, in XVFI demo, the subjective quality for the two clips they show look amazing as compared to DAIN and other works. So, wanted to know if you have also tested on similar sequences for SoftSplat and how does those sequences look.

sniklaus commented 3 years ago

Every paper has a demo where the proposed result works better than the others. :slightly_smiling_face:

If you have a specific sample in mind, I am happy to run SoftSplat on it and post the result here.

rsjjdesj commented 3 years ago

Every paper has a demo where the proposed result works better than the others. 🙂

If you have a specific sample in mind, I am happy to run SoftSplat on it and post the result here.

Sure. I understand. I will see if I can get some sample images.

There is also a separate thread going on https://github.com/JihyongOh/XVFI/issues/5. @sniklaus : please check that thread, and those images and let us know your opinion.

sniklaus commented 3 years ago

I shared my thoughts in the reference thread/issue. :+1: