kaancolak
commented
1 month ago I developed two prototype models using PointNet and PointNet++. Each model is a combination of two sub-models: a regression transform net to estimate the object's center and a PointNet-based network to predict bounding box dimensions and yaw angles. The bounding box estimation part from the Frustum-PointNet paper was used as a reference.
The models take point clouds and labels as inputs.
The PointNet-based model can process around 2,000 objects using pure Python and a 3060 mobile GPU. By using TensorRT quantization and a more powerful GPU, it can process over 4,000 objects without performance issues. It looks applicable to Autoware.
Although the PointNet++ model is slightly more accurate, it is only half as fast. Pure Python version processing ~900 objects each second.
I will share based models that were used in the experiments after small refactoring. I used NuScenes 3D labels as a train and validatioın purpose.
I shared the evaluation results of the based model below;
Test metrics results: BEV Intersection over Union 2D: 0.8705733457868103 Intersection over Union 3D: 0.7107460938907424 Intersection over Union 3D > 0.7: 0.6023422951582867
Initial results:
FYI: @xmfcx @miursh @shmpwk @OsamuSekino
xmfcx
Checklist
Description
In the current version of Autoware, the L-shape fitting algorithm is used to find 3D bounding boxes for vehicles from object clusters. However, this algorithm has some limitations for accurately identifying these bounding boxes. If the vehicle's point cloud includes side mirrors, antennas, or only shows the back of the vehicle (forming an I-shape), the algorithm often fails. Additionally, it can struggle in situations where there are only a few points available.
The Shape-fitting algorithm has crucial roles especially when using camera-lidar fusion or using Apollo instance segmentation as an object detector.
L shape fitting results :
https://github.com/autowarefoundation/autoware.universe/assets/12658936/fef4f1ae-b364-473e-aaa4-b55e6eb69f30
Purpose
Add a neural network-based 3D bounding box shape estimation algorithm. This could be an optional feature in the shape_estimation package, allowing users to choose it if desired.
Possible approaches
The point-based method can be used for estimating object bounding boxes. Typically, networks like PointNet/PointNet++ are used for classifying point clouds or performing point-wise instance segmentation, but they can be easily adapted to estimate 3D bounding boxes of objects.
Definition of done
[ ] - Train and evaluate prototype model [ ] - Prepare the TensorRT inference engine and update shape_estimation package [ ] - Add model to the model zoo and prepare documentation about how to train