search

markisus / pytagmapper

Python Mapping library for Fiducial Tags
MIT License
19 stars 2 forks source link

camera distortion #7

Open dcnieho opened 7 months ago

dcnieho commented 7 months ago

Hi, I find this work very interesting, as i wish to figure out exactly where my markers are on a non-coplanar ArUco board. I am looking through the code, and am wondering the following:

  1. I see you use the camera_matrix in calls to cv2.solvePnP() but not distortion coefficients. Should i undistort my images before using them with the map builder?
  2. I also see that your project function doesn't take distortion parameters. For the cv2.solvePnP() calls i could just add them, for this function as you wrote it yourself its more complicated. Could it be replaced with OpenCV functions like cv2.projectPoints() so that distortion coefficients can be used?

Thanks!

dcnieho commented 7 months ago

Ah, first paragraph of your readme :p

dcnieho commented 7 months ago

Super cool stuff, got it to work. Do you welcome pull requests? I changed show_map.py to use a 3D plot when plotting a 3D map, can send that your way.

dcnieho commented 7 months ago

Ok, and a last question. For a single marker, cv2.solvePnP() has an ambiguity (see, e.g. https://github.com/opencv/opencv/issues/8813). One can get both solutions by calling ret, rvecs, tvecs, reprojectionError = cv2.solvePnPGeneric(objectPoints, imgPoints, cameraMatrix, distCoeffs, flags=cv2.SOLVEPNP_IPPE_SQUARE), whereas cv2.solvePnP() only returns the "best" one of the two. Would the built map be better if both solutions are taken into account, and consistency in poses between markers in the various images used to prune the wrong solutions returned by the PnP solver? I don't yet understand the logic and math of the method code, so am not able to help.

markisus commented 7 months ago

Hi @dcnieho, sorry for the late response. I'm glad you got everything to work!

Yes a pull request is welcome. The only concern is if it pulls in a massive dependency on a new third party library.

As for your second question, I'm not sure. When building maps, an image with a single marker contributes no information to the map. For live inside out tracking, you may end up with a camera frame with just a single marker (or no markers) during camera movement. In these scenarios, it could be a good idea to try both solutions and see which one is more consistent with the previous known camera pose.