6D Object Localization/3D Object Detection Models

[JisuHann]

Datasets

Trend

1. detect 2D targets first -> solve their 6d poses with a PnP-algorithm
   • Approach to use pixel-wise RANSAC-based voting scheme is time-consuming
   • Approach to first localize and crop the bounding boxes of the objects of interest
2. solve their 6d poses with a PnP-algorithm
   • runtime increase bc need to compute the 6D pose via PnP

With Deep Neural Network

1. EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach 논문 code
   • RGB data / custom dataset에 대한 내용이 따로 없음
2. G2L-Net: Global to Local Network for Real-time 6D Pose Estimation with Embedding Vector Features(CVPR 2020) 논문 code
   • custom dataset에 대한 내용이 따로 없음
3. HybridPose: 6D Object Pose Estimation under Hybrid Representations(CVPR 2020) 논문 code
   • 읽어봐야 알겠지만 뭔가 RGB data같음/ custom dataset에 대한 내용이 따로 없음
4. PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation 논문 code
   • RGBD data
5. Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation 논문 code
   • RGB data

Without Deep Neural Network

1. Point pair feature: Model Globally, Match Locally: Efficient and Robust 3D Object Recognition (CVPR 2010) 논문

[JisuHann]

EfficientPose: An efficient, accurate and scalable end-to-end 6D multi object pose estimation approach

Problem

• On 6D object pose estimation task using RGB input (especially on LineMOD)
• most of the recent works of [6D object pose estimation with RGB input] => Two stage
  1. detect 2D targets first -> solve their 6d poses with a PnP-algorithm
     • Approach to use pixel-wise RANSAC-based voting scheme is time-consuming
     • Approach to first localize and crop the bounding boxes of the objects of interest ->the whole method needs to be applied for each detected object separately
  2. solve their 6d poses with a PnP-algorithm
     • runtime increase bc need to compute the 6D pose via PnP

Goal

• predict 2D object detection and 6D poses within a single shot
  • on 1-2 problem, extend 2D object detection architecture family EfficientDet to predict the 6D poses of objects -> add two extra subnetworks to predict the translation and rotation of objects, analogous to the classification and bounding box regression subnetworks
  • no need of further post-processing steps like RANSAC and PnP

In this paper..

• 6D pose: 3D rotation of the object and the 3D translation --> add two new subnetworks
• (classification subnetwork->) Rotation subnetwork
• (bounding box regression subnetwork->) Translation subnetwork and use additional anchor box mapping and non-maximum-suppression of the base architecture (to filter out background and multiple detections)

  finally, this willl detect these in a single shot

• class (class net)
• 2D bounding box (box net)
• Rotation (rotation net)
• Translation (translation net)

with scaling hyperparameter Phi

Rotation Network

• output: one rotation vector for each anchor box
• not using the output directly as the regressed rotation, but add an iterative refinement module
  1. initial regression
  2. iterative refinement

Translation Network

• output: a translation for each anchor box
• instead of directly regressing all components of the translation vector, but we split the task into predict the 2D center point of the object in pixel coordinates and the distance separately

Additional

1. Transformation Loss
2. 6D Augmentation
3. Color space Augmentation

[JisuHann]

Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation

Problem

• On 6D pose Estimation task (Challenging)
  • problems such as occlusion and symmetries
  • difficult to construct 3D models with precise texture without expert knowledge or specialized scanning devices
• difficulties on the textured 3D model of an object

Goal

• Predict 3D coordinates pixel-wise prediction -> used to form 2D-3D correspondences
• predict pose without textures on 3D models (bc infinite and continuous symmetric poses on a single axis)
• robust to occlusion
• make a novel transformer loss(novel loss function) to handle symmetric objects by guiding predictions to the closest symmetric pose (especially, each pixel forms a 2D-3D correspondence that is used to compute poses by the PnP algorithm)

In this paper...

1. RGB Input + 2D Detection(Bounding box)
2. Mask prediction & Bbox adjustment
3. 3D coordinate & Error prediction
4. PnP, RANSAC -> Output: Estimated poses

[JisuHann]

Dataset

BOP Challenge dataset

1. Linemod

2. YCB-Video

   002_master_chef_can 003_cracker_box 004_sugar_box 005_tomato_soup_can 006_mustard_bottle 007_tuna_fish_can 008_pudding_box 009_gelatin_box 010_potted_meat_can 011_banana 019_pitcher_base 021_bleach_cleanser 024_bowl 025_mug 035_power_drill 036_wood_block 037_scissors 040_large_marker 051_large_clamp 052_extra_large_clamp 061_foam_brick

3. SMOT

   crackerbox bleach driller mustard pitcher tomato can maxwell can sugar box

4. T-Less

5. HomebrewedDB

• YCB-Video 데이터가 효율적이다.