Object detection algorithm

[Supernova1114]

URC 2024 Rules In this year's autonomy mission, there are requirements for object detection other than Aruco tags. We will need to detect an orange mallet, and a plastic reusable water bottle. You can find specific in the rules including a link to the mallet item.

• [x] Write description of plan for training, frameworks used, etc.
• [x] Test prebuilt model.
• [x] Train model for mallet detection.
• [x] Train model for water bottle detection.
• [ ] Get model working with Isaac ROS object detection??

[Supernova1114]

Some object detection libs:

• https://github.com/google/mediapipe
• OpenCV
• ZED sdk
• Yolo v8

[Supernova1114]

Update:

• Nvidia has object detection packages in the Isaac ROS library. Support for Yolov8 and detectnet model.
• Support for ROS
• Hardware accelerated
• https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_object_detection

[Supernova1114]

ToDo:

• [x] Order orange mallet
• [x] Order extra water bottles / colors for training.

[Supernova1114]

Doing research:

• https://www.amazon.com/s?k=32+oz+water+bottles&crid=39JBB2NLSMH04&sprefix=32+oz+water+bottle%2Caps%2C139&ref=nb_sb_noss_1
• https://www.kaggle.com/code/abdallahwagih/bottle-synthetic-classification-acc-99
• https://www.youtube.com/watch?v=m9fH9OWn8YM
• https://www.kaggle.com/code/abdallahwagih/water-bottle-efficientnetb3-85-71
• https://storage.googleapis.com/openimages/web/index.html

[kavehzare98]

Progress Report:

• We have a working model that can identify most objects (specifically bottles) based on camera feed.
• Current issue: pre-trained model cannot identify hammers or the orange mallet.
• Able to calculate distance, but error grows with distance. Planning on using a distortion matrix to reduce error plus using a camera with a known focal length. (1/12/24)
• LATEST UPDATE (1/21): Trained model on 1900 annotated images of orange mallet (with few bottle annotations). ~4500 images post-augmentation. Using Google Colab's A100 GPU. Able to use 0.7 confidence with approx. 80-90% precision. Current training duration: 2 hours
• 389/500 compute units left on Colab with subscription

Things To Do:

• [x] Gather our own images on Saturday, January 13th, 2024 for the orange mallet (10,000 images) and water bottles (5,000 images).
• [x] Annotate our dataset via Roboflow by January 16th.
• [x] Train model on custom dataset on Google Colab
• [ ] Test model on orange mallet and Nalgene bottles in appropriate environment using ZED
• [ ] Configure Object's distance from rover via bounding boxes and frame coordinates
• [ ] ROSify and test on rover
• [ ] (Optional) Configure obstacle avoidance
• [ ] Looking into Ensemble where we get to simultaneously use two separately trained models for bottles and mallet.

[kavehzare98]

Configuring Object's distance from sensor (camera) - current research:

• https://stackoverflow.com/questions/14038002/opencv-how-to-calculate-distance-between-camera-and-object-using-image

• https://github.com/opencv/opencv/blob/4.x/samples/python/calibrate.py

• https://www.geeksforgeeks.org/realtime-distance-estimation-using-opencv-python/

• https://pyimagesearch.com/2015/01/19/find-distance-camera-objectmarker-using-python-opencv/

Formula Used to Find Distance:

• F = (P x D) / W => Solve for D' = (F x W) / P

F is the Focal Length of Camera

measure the apparent width in pixels P

distance D from our camera

a marker or object with a known width W

[Supernova1114]

Labeling is in progress.

[Supernova1114]

Object detection implementation is done, including a distance estimator.