Python bindings for apriltag_detect throws error when no detection is found

[nlbutts]

Describe the bug See the example Python code below. If an AprilTag is in the FOV and detected, everything works great. If no AprilTag is in the FOV, the Python bindings throw the following error:

Traceback (most recent call last):
  File "/home/nlbutts/apriltag_test.py", line 30, in <module>
    dets = detector.detect(gray)
           ^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Unable to create 1 threads for detector

To Reproduce Steps to reproduce the behavior: Run the program below with the AprilTag in view, then cover the lens and it stops working.

Expected behavior A clear and concise description of what you expected to happen. It should return 0 detections not crash.

Input Image If applicable, please attach the input image that reproduces the problem. This can be re-created at will.

Screenshots If applicable, add screenshots to help explain your problem. See above.

Operating Sytem PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Installation Method

git clone <apriltag repo>
cd <repo>
mkdir build
cd build
cmake ..
make -j4
sudo make install
export PYTHONPATH=/usr/lib/python3.11/site-packages
python <program below>

Cover camera and BAM.

Code version Using this hash: 77a769ac9eb4205e6518e60a2c48ae278cdf9612

Additional context Add any other context about the problem here.

import apriltag
from picamera2 import Picamera2
import cv2 as cv
import numpy as np

'''
Camera cal data
camera matrix:
 [[1.52373952e+03 0.00000000e+00 8.10534467e+02]
 [0.00000000e+00 1.55178225e+03 3.03966879e+02]
 [0.00000000e+00 0.00000000e+00 1.00000000e+00]]
'''

detector = apriltag.apriltag(family='tag36h11')

picam2 = Picamera2()
picam2.configure(picam2.create_video_configuration(main={"size": (1280, 720), "format": "BGR888"}))
print(picam2.camera_config)
picam2.start()

count = 0

while True:
    print(f'Image {count}')
    count += 1
    img1 = picam2.capture_array()
    h1, w1, ch = img1.shape
    # AprilTag dections
    gray = cv.cvtColor(img1, cv.COLOR_BGR2GRAY)
    dets = detector.detect(gray)
    if dets is not None:
        print(dets)
    # for det in dets:
    #     print(det)
        # corners = np.round(np.array(det['lb-rb-rt-lt']), 0).astype('int')
        # cv.line(img1, (corners[0][0], corners[0][1]), (corners[1][0], corners[1][1]), (0, 255, 0), 2)
        # cv.line(img1, (corners[1][0], corners[1][1]), (corners[2][0], corners[2][1]), (0, 255, 0), 2)
        # cv.line(img1, (corners[2][0], corners[2][1]), (corners[3][0], corners[3][1]), (0, 255, 0), 2)
        # cv.line(img1, (corners[3][0], corners[3][1]), (corners[0][0], corners[0][1]), (0, 255, 0), 2)
        # center = np.round(det['center'], 0).astype('int')
        # id = det['id']
        # txt = f'{id}'
        # cv.putText(img1, txt, (center[0], center[1]), cv.FONT_HERSHEY_SIMPLEX, 1.5, (255, 255, 255), 2)

[christian-rauch]

I adapted your script to use OpenCV's VideoCapture class:

#!/usr/bin/env python3
import apriltag
import cv2

if __name__ == "__main__":
    cam = cv2.VideoCapture()
    if not cam.isOpened():
        exit()

    detector = apriltag.apriltag(family='tag36h11')

    count = 0

    while True:
        _, image = cam.read()
        cv2.imshow("image", image)
        cv2.waitKey(1)

        # print(f'Image {count}')
        count += 1

        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        dets = detector.detect(gray)
        print(f"image {count}, detections", len(dets))
        if len(dets) > 0:
            print(f"detections", dets)

With the "Tag36h11" from https://april.eecs.umich.edu/media/apriltag/tagformats_web.png moving in and out of the camera I get a log like:

image 38, detections 0
image 39, detections 0
image 40, detections 0
image 41, detections 0
image 42, detections 1
detections ({'hamming': 1, 'margin': 95.90980529785156, 'id': 0, 'center': array([217.63509135, 132.08676829]), 'lb-rb-rt-lt': array([[167.09477234, 163.29779053],
       [252.62365723, 179.94235229],
       [266.43988037, 101.9475174 ],
       [183.34217834,  85.18266296]])},)
image 43, detections 1
detections ({'hamming': 0, 'margin': 105.45146179199219, 'id': 0, 'center': array([223.41219345, 125.66384523]), 'lb-rb-rt-lt': array([[171.00808716, 157.98808289],
       [259.49859619, 175.00976562],
       [274.39813232,  94.21437073],
       [187.78811646,  76.95012665]])},)
image 44, detections 1
detections ({'hamming': 0, 'margin': 107.93148040771484, 'id': 0, 'center': array([227.28037937, 119.98327687]), 'lb-rb-rt-lt': array([[173.24375916, 153.52482605],
       [264.44946289, 171.06555176],
       [279.59329224,  87.51166534],
       [190.35250854,  69.2325058 ]])},)
[...]
image 74, detections 1
detections ({'hamming': 0, 'margin': 109.63236236572266, 'id': 0, 'center': array([186.29173532, 141.22647609]), 'lb-rb-rt-lt': array([[125.28643799, 190.19546509],
       [235.91940308, 200.60371399],
       [246.70794678,  92.73034668],
       [136.07653809,  81.14628601]])},)
image 75, detections 1
detections ({'hamming': 0, 'margin': 103.26177215576172, 'id': 0, 'center': array([178.18746309, 147.52838709]), 'lb-rb-rt-lt': array([[119.40704346, 196.05543518],
       [225.92948914, 205.7492218 ],
       [235.50740051, 100.20706177],
       [130.6658783 ,  89.57637787]])},)
image 76, detections 0
image 77, detections 0
image 78, detections 0
image 79, detections 0
image 80, detections 0
image 81, detections 0

For me, the Python bindings work even when the detections disappear. Are you seeing this issue only on the Raspberry Pi? Can you reproduce this with my adapted script on a different PC?

[nlbutts]

The python bindings worked fine with OpenCV or Pillow. But when I started a picamera2 capture, I would always get the error with spinning up threads. Although it did seem to process one frame, but the second frame would throw this error message. I write a small shared library that made the calls to the apriltag library and then used ctypes to expose the interface to my new shared library. That worked fine. I dug through the apriltag python interface, I don't see why it would fail to spin up the threads. When the FRC session is done, I might take a deeper look into this. Really weird.

[christian-rauch]

I can reproduce this now. However, I am sure this has nothing to do with the threads, but rather with something setting errno to EAGAIN.

If I add something like perror("ERR"); at the beginning of apriltag_detect, then this will already print No such file or directory for the first image (Image 0) and Resource temporarily unavailable for the next image (Image 1) without actually running any detection. So I would assume that the errno is set internally by something in capture_array().

You can work around this by printing something right after capture_array(), e.g.:

    img1 = picam2.capture_array()
    print("")

will reset the errno and thus print ERR: Success.

Maybe just resetting the errno at the beginning of apriltag_detect would be a good way to make sure that we only consider errnos generated from function calls inside apriltag_detect. But I also think that picamera2 should do better error checking.

[christian-rauch]

@nlbutts Can you check if https://github.com/AprilRobotics/apriltag/pull/323 fixes your issue?

[nlbutts]

That fixed the problem. So did the print("").