Problems with DrawAxis function

[tas-pats]

System Information

OpenCV version: opencv-python 4.7.0.72 and opencv-contrib-python 4.10.0.84 OS: Windows 11 Python version: 3.11.3

Detailed description

I am detecting the ChArUco boards using the OpenCV. Once I have detected and estimated its pose, I am trying to plot the axis using the drawFrameAxes function; however, I can see different axis plotted just by changing their lengths even and everything remains the same. The included photo has two axis of different length plotted:

Steps to reproduce

original image:

import os
import numpy as np
import matplotlib.pyplot as plt
import cv2

# camera params after calibration
mtx = np.array([[3341.0930071 ,    0.        , 2020.04757797],
   [   0.        , 3332.54238818, 1108.10670945],
   [   0.        ,    0.        ,    1.        ]])

dist = np.array([-0.10474984,  0.04565679,  0.00948721,  0.        ,  0.        ])

img_rgb = cv2.imread("some img")
img_rgb = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2BGR)
img = cv2.cvtColor(img_rgb, cv2.COLOR_RGB2GRAY)

# board info
dictionary = cv2.aruco.getPredefinedDictionary(cv2.aruco.DICT_4X4_100)
board = cv2.aruco.CharucoBoard((7, 5), 0.055, 0.041, dictionary)

params = cv2.aruco.DetectorParameters()
detector = cv2.aruco.ArucoDetector(dictionary, params)

# detect markers and charuco corners
marker_corners, marker_ids, rejectedCandidates = detector.detectMarkers(img)

ret, charucoCorners, charucoIds = cv2.aruco.interpolateCornersCharuco(marker_corners, marker_ids, img, board)

ret, rvec, tvec = cv2.aruco.estimatePoseCharucoBoard(charucoCorners, charucoIds, board, mtx, dist, None, None)

# start plotting
axis_size = 0.1
result_short = cv2.drawFrameAxes(np.copy(img_rgb), mtx, dist, rvec, tvec, axis_size)

axis_size = 5
result_long = cv2.drawFrameAxes(np.copy(img_rgb), mtx, dist, rvec, tvec, axis_size)

# plot
plt.subplot(1,2,1)
plt.imshow(result_short)
plt.subplot(1,2,2)
plt.imshow(result_long)

plt.show()

result_both = cv2.drawFrameAxes(np.copy(img_rgb), mtx, dist, rvec, tvec, 0.2)
result_both = cv2.drawFrameAxes(result_both, mtx, dist, rvec, tvec, 2)

### Issue submission checklist

- [X] I report the issue, it's not a question
- [X] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
- [X] I updated to the latest OpenCV version and the issue is still there
- [X] There is reproducer code and related data files (videos, images, onnx, etc)

[A-Choudhari]

thats strange. I replicated it on my machine but everything seems to be fine. both lines seem to have the same direction. please let me know if I'm misunderstanding the issue or anything.

[tas-pats]

@A-Choudhari if you have included your generated image (which looks like you did since I didn't change opacity) you should see that your lines have the almost the same direction, but why there is a (small) gap between the two when parameters are the same and the only difference is their length? I would expect both vectors to be following exactly the same path, but one would be longer. If you wanted to see the green (y-axis) point in a different direction, try extending the axis_size to a larger value, such as 10+.

[A-Choudhari]

I tested it out with various axis sizes now and what I noticed is that as long as the ratio between the two axis sizes are less than or equal to 3, the difference in the direction of the vectors is not noticeable. Also, if the axis-size values are both less than 1, I wasn't able to pick up any difference from the eye, but logic-wise there probably is some disparity. I'll start looking into this and try to fix it. The issue must be due to rounding errors. Here's some additional figures that I tested out.

[crackwitz]

I'm guessing it could be an issue with the drawing primitives. you should drastically shorten your axes so their end points stay within view.

Context from Stack Overflow: https://stackoverflow.com/questions/78808361/problems-with-opencv-draw-axis-function

[crackwitz]

Yep, it's definitely an issue with the line() primitive and how it (badly) handles end points being too far out of the bounds of the image. if there is any clipping going on, it's not implemented properly.

this has been an issue for years.

IDK if anyone will touch the drawing primitives in imgproc. everyone hopes that with OpenCV 5 we'll get something based on OpenGL. I do anyway.

[Octopus136]

I seem to have had a similar problem before.

When dealing with this kind of perspective, if a line is too long in length, it may cross the end point, resulting in being mapped in the opposite direction.

Of course, this is just my personal guess.

[Octopus136]

The drawFrameAxes function creates four points (0, 0, 0), (length, 0, 0), (0, length, 0) and (0, 0, length) in 3D space and maps them to the 2D image according to the given parameters, which is accomplished by the projectPoints function.

However, due to camera distortion, it is normal behavior that points of different lengths do not necessarily end up on a vector. However when the length is too long, the possibility of the line transforming to other directions is indeed a problem worth considering, which could be a problem caused by the end point of the line crossing the end point of the perspective as I mentioned above.

I haven't done any experiments on this, and I'm not an expert in the field. This is a guess based on my previous experience of doing a little work related to perspective transformations.

[A-Choudhari]

Yes, I have been noticing the same thing. For now, the best fix would be to make any line that extends out of the frame of the picture to be changed into a new line with a shorter length that is till the end of the frame. This check will have to be done for all 3 lines. I have started basic implementation locally but still have a bit of work to do.