search

hzwer / Practical-RIFE

More practical frame interpolation approach.
MIT License
623 stars 68 forks source link

Can RIFE fix time parallax (rolling shutter) effect? #103

Closed ioctl-user closed 2 months ago

ioctl-user commented 2 months ago

Which model is the best to fix rolling shutter effect in live video, if any?

If it's not, what could you recommend for this problem?

flowreen commented 2 months ago

ChatGPT said no. "Since rolling shutter is a spatial distortion caused by the sensor, RIFE may not be able to directly fix these artifacts because it deals with temporal interpolation between frames, not spatial correction within individual frames." https://chatgpt.com/share/66e5e70e-9260-800b-a1ba-86483eec1810

ioctl-user commented 2 months ago

It would appear, that RIFE is very close to solve this problem, because rolling shutter fix is the same as frame interpolation with time shifts of parts of the frame relative to each other.

I.e, simple frame interpolation works like this (vertical is frame content, horizontal is time of frame capture):

|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |         to     |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |

While rolling shutter fix with frame interpolation should works like this:

|       |       |       |       |       |                |   |   |   |   |   |   |   |   |   |   |
 |       |       |       |       |       |               |   |   |   |   |   |   |   |   |   |   |
  |       |       |       |       |       |              |   |   |   |   |   |   |   |   |   |   |
   |       |       |       |       |       |      to     |   |   |   |   |   |   |   |   |   |   |
    |       |       |       |       |       |            |   |   |   |   |   |   |   |   |   |   |
     |       |       |       |       |       |           |   |   |   |   |   |   |   |   |   |   |
      |       |       |       |       |       |          |   |   |   |   |   |   |   |   |   |   |
       |       |       |       |       |       |         |   |   |   |   |   |   |   |   |   |   |
hzwer commented 2 months ago

You may check these works. https://scholar.google.com/scholar?hl=zh-CN&as_sdt=2005&sciodt=0%2C5&cites=12783272439091205239&scipsc=1&q=rolling+shutter&btnG=

I notice a paper named Rolling Shutter Correction with Intermediate Distortion Flow Estimation

ioctl-user commented 2 months ago

You may check these works. https://scholar.google.com/scholar?hl=zh-CN&as_sdt=2005&sciodt=0%2C5&cites=12783272439091205239&scipsc=1&q=rolling+shutter&btnG=

I notice a paper named Rolling Shutter Correction with Intermediate Distortion Flow Estimation

Seems, there is no production-ready projects.

Thank you for answer.