search

hello-robot / stretch_ros

ROS packages for the Stretch mobile manipulators from Hello Robot Inc.
https://docs.hello-robot.com/0.2/stretch-tutorials/ros1/
174 stars 86 forks source link

Fixed aruco tag detection for latest version opencv #100

Closed sylvia-segfault closed 11 months ago

sylvia-segfault commented 1 year ago

We used aruco.ArucoDetector from the latest version of opencv to create a detector object, which is saved in self.detector. This detector object is then used to call drawDetectedMarkers and detectMarkers as shown in line 542 and line 559.

The main changes we had to make were for estimatePoseSingleMarkers inside the update function of the ArucoMarker class. The official documentation says to use solvePnp because estimatePoseSingleMarkers was deprecated in the newest version of opencv. We manually created 3D points (line 196 - line 201) as the object points passed to solvePnP. We used the source code of estimateSinglePoseMarkers as a reference and found that estimateSinglePoseMarkers calls solvePnP and it used getSingleMarkerObjectPointers to calculate the object points. Thus, we used the source code of getSingleMarkerObjectPointers to compute 3D object points in Python.

We also created 3 dimensional tvecs and rvecs by calling solvePnP for every marker (for len(self.corners) amount of iterations) and appended the result of rvec_ret and tvec_ret to tvecs and rvec respectively (notice that these two are different from the final rvecs and tvecs since rvec_ret and tvec_ret are 2D vectors) returned by solvePnP at every iteration of going through all the markers.

hello-chintan commented 1 year ago

Hey @sylvia-segfault, thank you for creating this PR. I am assuming that the changes you have made are for the latest version of the package opencv-python as opposed to opencv-contrib-python which we use for importing the ArUco detection module and deep learning modules.

I suspect that merging this PR would break detection for users using the recommended opencv-contrib-python package as these packages are not API-compatible.

sylvia-segfault commented 1 year ago

Hi Chintan! Thank you for your response. @MarkusSchiffer and @vinitha910 tried to install the latest version of opencv-contrib-python but still produced errors. It sounds like we would need a very specific configuration for it to work. Could you share the branch of the repo you guys used to run the ArUco markers as well as the version of opencv-contrib-python you used? Thank you!

hello-chintan commented 1 year ago

@sylvia-segfault, the recommended version is opencv-contrib-python >= 4.7.0 along with the dev/noetic branch. If you also have an opencv-python module installed, be sure to uninstall it as it could result in a cv2 namespace collision. Finally, it's a good idea to check the version of opencv being imported in python by launching the IPython interpreter in a terminal and running the following statement: import cv2; cv2.__version__ This should print '4.7.0'. Please report back what you see after running the same.

hello-binit commented 11 months ago

Thank you for opening this PR @sylvia-segfault! We had previously pinned OpenCV to an older version, which explains why we couldn't merge this earlier. I just updated the version of OpenCV to 4.8 and the Aruco API has changed as you described. You were ahead of the curve here! Thanks again for filing this fix!