Video Stabilization problem

[mppcasc]

Hi, I used the result of calibration toolbox to do video stabilization for the video in the paper. I found that according to the time offset in the calibration result, the stabilized video was very bad. After scaling and offsetting the gyroscope data(as follow), I can get satisfactory results.

coeffies = [1.60190524e-03, -2.79751510e+00] # rotation.mp4 gyro_times = (gyro_times + coeffies[1]) / (1 - coeffies[0])

My question is, the video stabilization in the paper mentioned in the Readme directly use the time offset( 2.8333333333333335 as follow) of the calibration toolbox?

rotation.mp4 calibration result by the calibration toolbox: gyro_rate = 853.660712254363 time_offset = 2.8333333333333335 gbias_x = -0.01001219994597313 gbias_y = -0.008216488048614853 gbias_z = 0.010610096301797737 rot_x = 0.022640457925138826 rot_y = -0.0068401509599219324 rot_z = -1.5426536394739456

Looking forward to your reply.

[hovren]

The values used in the paper should be those that are stored in the <video_name>_reference.csv file.

You can't look at the time offset value alone since the full time synchronization is using both the offset and the estimated IMU sample rate. Your affine transform of gyro_times using coeffies is correcting for an incorrectly estimated gyro_rate and time_offset.

The calibration result is also not deterministic, so running it again might produce a different set of calibration parameters.

Since your offset is in the order of one image frame it is also possible that whatever external program you are using to produce the stabilized video has an indexing-off-by-one-error somewhere.

[mppcasc]

Each calibration result is different because different slices are used for optimization each time. From my observations, the time offset is almost the same each time. There may be small differences in the gyro bias and three axis data, but they do not seem to have much effect on the stability results. The result of time offset is not accurate, which has great influence on the final result. Is this phenomenon true?

[hovren]

Yes, the timing parameters are probably more important than the other parameters in general. As you can see in the paper we sometimes observed bad results on the RC-car due to the combination of timing errors and the high frequency motions in that sequence. The rotation sequence however always produced nice video results, which is why I am a bit surprised that it fails for you.

I still suggest you check the code for the program you use to stabilize the video using the calibrated parameters for any errors. Have you tried using the values in the rotation_reference.csv file? Because those values are chosen precisely because they produced nice stabilized videos.

Unless you are using exactly the same versions of numpy and scipy that I used in 2015 (which I doubt :)) there is also a slight risk that something has changed which produces this error.

[mppcasc]

Hi, now my video stabilization program works, thank you. The reason it didn't work before is that I did rotation compensation on the first frame of the video, whereas it didn't need to do anything on the first frame and did rotation compensation from the second frame.

The smoothing method I just used is like scipy.signal.sosfiltfilt method, which uses a smoothness factor to interpolate the quaternions forward and then backward to get the final smoothing sequence. When the smoothness factor is relatively large, some slices (located at 4 seconds of rotation.mp4) in the video cannot be perfectly corrected. With the reduction of the smoothness factor, a perfect effect can be achieved. Have you ever come across this phenomenon?

[hovren]

Glad to hear that you found your problem. I'm closing this issue.

And no, we did not do more regarding stabilization than the simple method that we describe in the paper.