Option for underwater stabilization

[stigla610]

Is there an easy way to make an option for underwater stabilization? I printed and laminated the calibration target and made lens profiles for most Hero 9 lenses but I think it would be relatively easy to add an Underwater checkmark (or option to set refraction index, water is 1.333) that would work for all the profiles in the database.

[AdrianEddy]

I don't think there's any easy way to calculate that, other then really running calibration under water. Do you know any algorithms, or math for that?

[stigla610]

Unfortunately, I don't. I'll definitely let you know if I find something

[anton-belkin]

I am also interested in the feature and can help with the math. But I need a headstart in understanding lens profiles in gyroflow. What are Pixel focal length (x,y), Focal center (x,y) and distortion coefficients (a,b,c,d)? I can guess what is focal center, but I have no ideas what are the others. How are pixel focal length and distortion coefficients related to FOV or focal length expressed in mm?

I can tell how focal length in mm or FOV is affected by water FOVw = 2 arcsin (sin (FOVa/2) / 1.33 ), where FOVa is FOV of the lens in the air. FOV is connected to focal length in a following way: FOV = 2 arctan(d/2f)/pi * 180, where f is focal length in mm and d is size of sensor in mm.

[AdrianEddy]

fx and fy are focal length values in pixels. f_mm = f_pixels sensor_width_mm / image_width_px f_pixels = f_mm / sensor_width_mm image_width_px

cx and cy is the principal point in pixels (usually width/2, height/2) These are part of the camera intrinsic matrix

More details about the lens model used in gyroflow: https://docs.opencv.org/4.x/db/d58/group__calib3d__fisheye.html

[AdrianEddy]

could you wrap your equations in ` so that github doesn't eat the asterisk?

[anton-belkin]

That is very helpful. Basically water refraction only affects pinhole projection, such that aw=tan(arcsin(sin(arctan(a))/k)) bw=tan(arcsin(sin(arctan(b))/k)) Where k = 1.33 is a refraction coefficient. and aw and bw are pinhole coordinates distorted by water.

But this is no linear transformation, its kind of additional distortion model, like fisheye of a kind.

So what could be done is after undistorting image into linear a,b coordinates apply the reverse transform from aw,bw into a,b (to undo water distortion) and then proceed as usual.

I can also write the water distortion in “clean” fisheye coordinates where u,v are arctan of an and b respectively (I.e. fisheye coefficients k1 to k4 are 0). They are also called spherical coordinates.

Uw= arcsin (sin(u)/k) Vw = arcsin (sin(v)/k)

I can also provide reverse transformations. Just let me know what helps you best.

[anton-belkin]

I will also try to approximate water distortion with distortion coefficients and see how precise is that.

[VladimirP1]

This r in the lens model is proportional to the sine of the angle of the incoming ray. So the change should be as simple as multiplying this r by the refraction index ratio in the distortion/undistortion code. So this is just a linear stretch of the undistorted image.

Note that this is not the same as changing the focal length, because focal length is stretching the distorted image.

[VladimirP1]

Of course assuming that the front lens is flat and perpendicular to the optical axis

[anton-belkin]

This r in the lens model is proportional to the sine of the angle of the incoming ray. So the change should be as simple as multiplying this r by the refraction index ratio in the distortion/undistortion code. So this is just a linear stretch of the undistorted image.

Note that this is not the same as changing the focal length, because focal length is stretching the distorted image.

I respectfully disagree. Since r is proportional to an and b, and a for instance is x/z = tan (angle of ray). It would have been proportional to sin if a was = x/sqrt(x^2+z^2)

[VladimirP1]

@antonbelkinlykke Ok, I agree, you are right

[AdrianEddy]

I can also provide reverse transformations. Just let me know what helps you best.

That would be helpful too

Thanks a lot, I'll check it out. Now I just need some underwater test footage, ideally with a checkerboard. @Aphobius maybe you have something useful?

[anton-belkin]

That would be helpful too

Actually they're almost the same.

a=tan(arcsin(sin(arctan(aw))*k))
b=tan(arcsin(sin(arctan(bw))*k))
(We just replace /k with *k)

[anton-belkin]

Okay, I think I figured it out. Let's look at effect of water in angle coordinates Theta_d.

Tw = arcsin(sin(T)*k)
Td = Distortion(Tw) ,where Distortion(T) = T*(1+k1*T^2+k2*T^4+k3*T^6+k4*T^8)

So I use Wolfram Alfa to compute Taylor Series of Distortion_w(T) = Distortion(arcsin(sin(T)*k))

Here is the result:

taylor series sin(arcsin(x)*1.33) at x =0
1.33 x - 0.17044 x^3 - 0.0616233 x^5 - 0.0340851 x^7 - 0.0223594 x^9

taylor series sin(arcsin(x)*1.33)^3 at x =0
2.35264 x^3 - 0.904471 x^5 - 0.211108 x^7 - 0.102017 x^9

taylor series sin(arcsin(x)*1.33)^5 at x =0
4.16158 x^5 - 2.66653 x^7 - 0.280666 x^9

taylor series sin(arcsin(x)*1.33)^7 at x =0
7.36142 x^7 - 6.60356 x^9

taylor series sin(arcsin(x)*1.33)^9 at x =0
13.0216 x^9

so, let's collect coefficients at powers of x:
q0 = 1.33
q1 = -0.17044 + k1*2.35264
q2 = -0.0616233 - k1*0.904471 + k2*4.16158 (ERROR FIXED 7.04.2024)
q3 = -0.0340851 - K1*0.211108 - k2*2.66653 + k3*7.36142
q4 = -0.0223594 - k1*0.102017 - k2*0.280666 - k3*6.60356 + k4*13.0216

These could be our new distortion coefficients, but we now have q0 for which there is no place in distortion model. Except there is - it should go to the next step and merge info focal length.

Let's now define new lens profile following way:

fxw = fx * q0
fyw = fy * q0
k1w = q1/q0
k2w = q2/q0
k3w = q3/q0
k4w = q4/q0
cxw = cx
cyw = cy

So, we can take an arbitrary lens profile and transform it for underwater this way. I have checked, precision of Taylor Series to x^9 is very high in this case, error is less than 1/10000.

[anton-belkin]

I will try to do that with my GoPro HERO12 and report what turned out to be the result.

This is computation using Google Sheets: https://docs.google.com/spreadsheets/d/15IOJ1NhFxSU4KMjk0xL4aEMORNOSOZK3cX0ftpkL72c/edit?usp=sharing

[anton-belkin]

Okay, I tried it. fx and fy I got right. It's now relatively stable at the center, all movements seem to come from translational movement of the camera. But that was easy part. But I got wrong distortion coefficients. When I replace profile coefficients with transformed, it becomes even more distorted than it was at the edges.

I will try simplier case of linear camera now.

[anton-belkin]

So,

First with GoPro's Wide FOV of about 90, nonlinearity of water distortion only reaches 2,5% at the edges, it happens to be both the case for linear (2.5% stretch) and fisheye lenses (2.5% shrink).

Therefore in the very first approximation, just multiplying focal length of the lens profile by factor 1.33 already gives very good results. Only sharp turns, 2.5% distortion is very slightly visible.

Then I also computed water distortion for a perfectly linear lens (which gopro linear almost is). The result is great. Here are distortion coefficients for linear lens distorted by water:

k1 | 0,4614834586
k2 | 0,1441834586
k3 | -0,04714977444
k4 | -0,124356391

I will experiment a bit more to find a mistake with general case of distortion coefficients calculation.

I can also provide underwater video files by GoPro12 in both Wide and Linear lens settings.

[VladimirP1]

This is my attempt at solving it numerically, but I have no underwater file to test. I had to make k0,k1,k2,k3 and focal length multiplier an optimization parameter. Error should be < 0.15 degrees almost everywhere except a couple of degrees near total internal reflection. https://colab.research.google.com/drive/1F0TsCPSY39q2_ihXNYRm2c41zAo4W2EF?usp=sharing

[anton-belkin]

Hey, thank you Vladimir! Here is a folder with two test videos: https://drive.google.com/drive/folders/1UWvaHtXhYJ9IwI4adKsFK_xFzbaLDDPb

One has Wide lens which is closer to fisheye. Another has linear lens.

Can you try and tell what are the results?

[VladimirP1]

Here is the modified profile for wide GoPro_HERO11 Black_Wide_8by7.json

[VladimirP1]

It seems to stabilize ok, but it is hard to tell because of huge translational motion. It would be easier if there were some far away objects in scene. By the way, I can access only one test video, for wide fov.

[anton-belkin]

Sounds great. What I was trying to achieve above is to provide to Gyroflow developers a formula that would algebraically convert existing lens profile into lens profile distorted by water. In such a case running numerical optimisation would be unnecessary for each case. I would appreciate if you could look at my post above to try and find a mistake in my logic or propose your own approach to it. https://github.com/gyroflow/gyroflow/issues/398#issuecomment-2041296453

I will also try to shoot a video with more distant objects underwater today.

[anton-belkin]

Here is the modified profile for wide GoPro_HERO11 Black_Wide_8by7.json

Hey, I just tried your profile myself - it's super steady. I can see that there's only translational movement left. May I ask you also make a profile for GoPro HERO12 Wide 16:9 please? I depart on diving tour in a couple of days and it would be so great to be fully ready for it with key stabilisation profiles. I am not sure I will finalise general formula so quickly...

And by the way, I have re-uploaded the second video file, so it should work now.

[VladimirP1]

@antonbelkinlykke Your approach kinda works too, just tried to repeat your steps in sympy to exclude human error. Only that it gives much higher angular error than least squares. Probably because taylor expansion is accurate only near the point (zero in this case).

This is a colab notebook with the process https://colab.research.google.com/drive/1XhchNDrYc5CSyZ_5dgd6-6NO3xkqXk7W?usp=sharing

[stigla610]

Hi guys, here is one of the videos I used to create a calibration profile for a Hero 9. Hope it helps. https://drive.google.com/file/d/1ysMIEd_Qq-9wHGiwih8jj5iwdLPuayve/view?usp=sharing

[VladimirP1]

GoPro_HERO11 Black_Wide_16by9.json @antonbelkinlykke Modified profile for 16:9, not tested of course

[VladimirP1]

But to get fully correct stabilization underwater I guess you would need to disable gopro's builtin rolling shutter correction, find the rolling shutter time for gopro 12 (no idea where, maybe @AdrianEddy knows?), enter it in gyroflow, select VQF integration method and use auto sync.

[AdrianEddy]

Yes, the rolling shutter correction is in SROT value in metadata (can use gyro2bb --dump to see it) and to disable internal rolling shutter correction it's oMBERS=0 in Labs

[anton-belkin]

GoPro_HERO11 Black_Wide_16by9.json @antonbelkinlykke Modified profile for 16:9, not tested of course

Thank you! I know about rolling shutter, to disable it on GoPro12 one has to download GoPro Labs firmware and use a designated QR code. But frankly, I do not see any impact of it so far. Maybe that’s due to long exposures in underwater darkness and thus relatively less impact of shutter.

[VladimirP1]

Maybe. But did you also use VQF/Complementary integration and sync manually and set non-zero rolling shutter time? That's needed because GoPro's pre-synced data is per-frame and not suitable for rolling shutter correction in post.

[AdrianEddy]

Wouldn't the underwater refractions make optical flow basically impossible to track things accurately? Making autosync unreliable

[VladimirP1]

At least on this sample autosync seems to give consistent results, but worse than on videos taken in air, seems to work properly only with large analysis duration like 1.5s, but that might be also due to me not actually knowing if this camera had internal rs correction disabled or not or maybe due to huge translation motion and not much rotational. What do you mean by refractions? These rapid changes in lighting caused by sunlight refraction while crossing air -> water? Is SROT in milliseconds?

[AdrianEddy]

What do you mean by refractions? These rapid changes in lighting caused by sunlight refraction while crossing air -> water?

Yes

Is SROT in milliseconds?

Yes

[anton-belkin]

Wouldn't the underwater refractions make optical flow basically impossible to track things accurately? Making autosync unreliable

I don’t think so. If you are not shooting through curvy moving water surface, which is not the case underwater, picture is distorted predictably. Talking about distorted sunlight playing on the bottom - yes, it adds a lot of movement, but not worse that tree leaves moving in the wind. Actually FCPX is dealing very well with stabilisation underwater, the issue with it is that it ignores lenses, thus adds distortion

[anton-belkin]

I have fixed an error in q2, now computation of general case formula should be super precise. Will check tomorrow.

[AdrianEddy]

btw, @antonbelkinlykke seems like you know a thing or two about lens distortions math, maybe you would like to take a look at #44 - there's a $1000 bounty for that, and we're basically missing the math for polynomial lens model

[anton-belkin]

GoPro_HERO11 Black_Wide_16by9.json

@antonbelkinlykke Modified profile for 16:9, not tested of course

I tested it today, works perfectly. Thank you very much!

I am still puzzled why my Taylor approximation does not work well for wide lens. Because it works perfectly for Linear lens. After I corrected an error in q2 it became much better for Wide, but still visibly worse than your least square optimised coefficients. I still hope to find a reason, for instance I suspect precision (number of digits) of coefficients given by free Wolfram Alfa online might be insufficient.

[VladimirP1]

This is what I get in that colab notebook I posted earlier. So it seems that you have some mistakes in signs in your formulas. And there's no need to divide by q0.

k_f 1.33000000000000
k_0 2.352637*k_0 + 0.1704395
k_1 0.90447129465*k_0 + 4.1615795893*k_1 + 0.1271487191975
k_2 0.790648104574771*k_0 + 2.66653212184398*k_1 + 7.36141813551277*k_2 + 0.110750293027951
k_3 0.765605672980835*k_0 + 2.67267847838483*k_1 + 6.60356013846174*k_2 + 13.0216125399085*k_3 + 0.106606636079901

[VladimirP1]

I do not think it possible to get as good results with taylor series, it is accurate only near zero. In my approach with least squares I am trying to minimize the error on the part of FOV that will actually be used by the camera (with some margin around total internal reflection, because without margin it seems that fisheye model does not approximate it well and error near the center increases) .

this is on same lens profile (wide 8:7): taylor:

least squares:

[VladimirP1]

I translated the least-squares approach to rust yesterday (depends on argmin and nalgebra) underwater2.zip

[anton-belkin]

@antonbelkinlykke Your approach kinda works too, just tried to repeat your steps in sympy to exclude human error. Only that it gives much higher angular error than least squares. Probably because taylor expansion is accurate only near the point (zero in this case).

This is a colab notebook with the process https://colab.research.google.com/drive/1XhchNDrYc5CSyZ_5dgd6-6NO3xkqXk7W?usp=sharing

@VladimirP1 you are right. I verified I have no computation errors. Taylor series approximation does not work with only 5 series members of this model. Least squares takes advantage of more freedom of choice in a way. @AdrianEddy the only feasible way we know now to compute water distorted lens profile is what Vladimir did with computational optimisation. I hope it's not difficult and you could add a feature to Gyroflow to automatically compute underwater profile, Vladimir proposes rust implementation above. Especially since it's not possible to compute such profile with a notepad or a spreadsheet. I would also like to ask if you can point me at piece of documentation explaining how to add my own profiles bypassing calibration, so that I can easily use profiles computed for GoPro by Vladimir.

[anton-belkin]

And yes, another probably more reliable approach could be using transformation directly, but that would require putting it in between pinhole model and distortion approximation theta = arcsin(sin(theta0)*k) and for backwards theta0 = arcsin(sin(theta)/k) And then having a parameter somewhere in UI. It could in theory be even keyframed, for mixed air/underwater footage cases. It would have been great to have it someday :) @AdrianEddy

[AdrianEddy]

what @VladimirP1 provided is ready to just plug into Gyroflow, I'll do it probably tomorrow. I'll add the keyframing too in the meantime, the #44 is still waiting to claim the bounty :)

[anton-belkin]

btw, @antonbelkinlykke seems like you know a thing or two about lens distortions math, maybe you would like to take a look at #44 - there's a $1000 bounty for that, and we're basically missing the math for polynomial lens model

Hey I left a comment there. Not sure if I got right what is a problem, that's left to be solved. Looked like so much has been done already.

[AdrianEddy]

I translated the least-squares approach to rust yesterday (depends on argmin and nalgebra)

Cool, looks like it works! Great job!

https://github.com/gyroflow/gyroflow/assets/3849067/45508442-7410-403f-9835-00617ac222f2

[AdrianEddy]

And there we have it: https://github.com/gyroflow/gyroflow/commit/c8038e69129e865b1733d31334f27589c08d9db4 @VladimirP1 any easy way to add the same for other lens models? I think the most useful would be Insta360

[AdrianEddy]

Dev build with this change is ready to use: https://gyroflow.xyz/devbuild/

[VladimirP1]

@AdrianEddy where can I find some sample values for k1, k2, k3, p1, p2, xi?

[anton-belkin]

And there we have it: https://github.com/gyroflow/gyroflow/commit/c8038e69129e865b1733d31334f27589c08d9db4

@VladimirP1 any easy way to add the same for other lens models? I think the most useful would be Insta360

Hey, I looked at your codebase here is a suggestion.

Why don't you add lrc as a an additional parameter to insta360 model and add following logic between line 31 and 33:

Let r = sqrt (x^2+y^2) Let z = r / tan (arcsin (sin(arctan(r/z)) * lrc))

I don't know rust syntax, but I used same variable names.

That would essentially do the refraction.

Note: approach with adjusting distortion coefficients was good, because it was inspiring. Whith no code changes we got to see that approach works. We also preserved the code structure you have. But, given how many models you have, and given the actual code structure as well, I think "in model" refraction computation might be more scalable as an approach. Just use the same (x,y,z) transformation in any other model. It would do the trick.

[anton-belkin]

Coefficient is 1.33 with the following digits depending on

1. water temperature
2. water salinity
3. light wavelength (colour) —> thus colour aberration at the edges

https://en.wikipedia.org/wiki/Optical_properties_of_water_and_ice#:~:text=The%20refractive%20index%20of%20water,loss%20at%20a%20particular%20wavelength.

Anton Belkin

On 9 Apr 2024, at 23:08, AdrianEddy @.***> wrote:

Dev build with this change is ready to use: https://gyroflow.xyz/devbuild/

btw, is the coefficient 1.33 or 1.(3)?

— Reply to this email directly, view it on GitHub https://github.com/gyroflow/gyroflow/issues/398#issuecomment-2045808455, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATZZ7SFFO2X6HNJFKSAJMQTY4QVC5AVCNFSM53FLWU6KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGU4DAOBUGU2Q. You are receiving this because you were mentioned.