search

yossato / virtualGimbal

virtualGimbal is an electronic stabilize device for videos that were taken by hand held camera. (I.e. DSLR)
162 stars 29 forks source link

How to remove the rolling shutter effect? #28

Open Fridaycoder opened 3 years ago

Fridaycoder commented 3 years ago

Hi, @yossato Very amazing work. I am trying to apply your program to mobile phones, now I have recorded the video and the corresponding gyro data, and run your code on my data, but the result looks not well. The main problem is the rolling shutter effect, which causes serious wobble distortion. I have tried running rolling_shutter_parameter_estimator to get the line_delay value, which I think is the parameter to deal with rolling shutter, but it has little effect, the wobble is still there.

Here are my steps: 1、Shooting the chessboard and log the corresponding gyro data; 2、Run ./camera_calibration to get the camera parameters; 3、Run ./rolling_shutter_parameter_estimator to get the line_delay value; 4、Shooting videos and run ./pixelwise_stabilizer to get the stabilized videos.

I think the steps are correct, but the result looks not as good as your demo. BTW, the direction of the gyroscope seems doesn't match any of your offered options since I have tried all the options and no result looks good :(

What can I do now?

yossato commented 3 years ago

Hi, @Fridaycoder Thank you for your comment. First of all, the rolling_shutter_parameter_estimator is incomplete. It does not give the correct values. I tried to find the coefficient of the rolling shutter with the software, but it did not work. So the coefficients used in the sample were measured by taking a video of a panel of LEDs blinking fast at about 1Khz, and measuring the scan time of a CMOS sensor by how many streaks appear on the screen. Also, I'm still finding things that can be improved in the rolling shutter removal algorithm. Specifically, the reference time of the gyro sensor needs to be the same for each line in a frame in the video, but this is slightly off for each line. I'm currently working on a branch to improve this. Also, if you don't turn off the built-in image stabilization function of the cell phone, it will interfere with the image stabilization function of VirtualGimbal.

Fridaycoder commented 3 years ago

Oh, I see. So the demo video just happens to have a good rolling shutter coefficient, and it just works? For the handheld devices, rolling shutter is a big problem and is very difficult to deal with :( BTW, I have already turned off the built-in image stabilization function of the cell phone and it shoots the unstabilized videos. Anyway, thanks for your reply, and hope you can make progress in this issue :)