search

ethz-asl / kalibr

The Kalibr visual-inertial calibration toolbox
Other
4.44k stars 1.4k forks source link

SolvePNP issue with BF fisheye lenses #594

Closed Hateley123 closed 1 year ago

Hateley123 commented 1 year ago

I am using Kalibr to calibrate 2 cameras in stereo with both using fish eye lenses and using the ds-none camera model. I am able to extract the intrinsic properties from the cameras but when the system tries to calibrate them in stereo I get this error: getTargetPoseGuess: solvePnP failed with solution: <sm.libsm.python.Transformation object at 0x7fo4cde41970> for all views. If i use extended unified camera model, this error only shows up for a few views, but the code is only able to process 61/313 views before the system crashes. Is there a different way I am supposed to use fisheye

Hateley123 commented 1 year ago

I have gotten it now where no matter what model I use I get the same error

NikolausDemmel commented 1 year ago

Kalibr does have some limitations for fisheye cameras. It can handle some distortion, but internally it will work with points on the normalized image plane, so if your FoV approaches 180deg, you run into issues.

Also the projection models that can handle FoV > 180deg in principle (like DS or Kannala-Brandt aka equidistant distortion) suffer from this issue in Kalibr. It might be that you are running into this?

You could give Basalt a try. It's not as feature-rich as Kalibr and the calibration process is a bit more manual, but does not have this limitation for fisheye models: https://gitlab.com/VladyslavUsenko/basalt/-/blob/master/doc/Calibration.md

Hateley123 commented 1 year ago

By the looks of our specs, we should be under the 180 deg threshold and I was able to calibrate the system before using kalibr. do you know what else could be causing the problem

NikolausDemmel commented 1 year ago

Not sure, sorry, could be something else of course. We did notice this well before 180deg, eg already with 150deg, but maybe it depends a bit on the lens itself and how good the initialization of lens parameters fits.

Hateley123 commented 1 year ago

Our lenses are a 130 and hers the weird thing, I can calibrate each camera indivdiually with no problems, but when I do multple topics then i get this error. was this ever come across before?

Hateley123 commented 1 year ago

for more info this is the error I get as well sometimes

WARN] [1674249977.632686]: Optimization diverged possibly due to a bad initialization. (Do the models fit the lenses well?) we have used litterally every model kalibr supports