search

maudzung / Complex-YOLOv4-Pytorch

The PyTorch Implementation based on YOLOv4 of the paper: "Complex-YOLO: Real-time 3D Object Detection on Point Clouds"
https://arxiv.org/pdf/1803.06199.pdf
GNU General Public License v3.0
1.21k stars 260 forks source link

More General View Adjusment of Point-cloud BEV #35

Open phuccuongngo99 opened 2 years ago

phuccuongngo99 commented 2 years ago

What?

This PR allows for the option to adjust point-cloud discretization in both x, y-direction from 3D point cloud data -> the bird-eye-view (BEV) image.

In other words, this PR allows users to adjust the boundary of BEV in both width and height directions. (See screenshots further down for more details)

The existing code works well for self-driving car well as for self-driving car, we are only interested in seeing what's in front of the car.

However, other applications of 3D point clouds object detection may have different settings such as industrial factories and warehouses like what I tried to do. We may need to adjust the view freely according to our set-up.

This PR works with existing code but also allows users to change if they wish. This PR doesn't break the original code or KITTI 3D Object Detection benchmark

Why?

The current code only allows for the adjustment in the forward direction.

Users can adjust the boundary of BEV image by setting the appropriate number of minX, maxX, minY, maxY here.

Scenario 1: Default Setting

The default is given here. https://github.com/maudzung/Complex-YOLOv4-Pytorch/blob/a712f52542a35893ddedbd7b782f34f31a0ff796/src/config/kitti_config.py#L13-L21

This is the BEV, notice how the view boundary starts from front of the car (it's because minX = 0) and we can see left and right view boundaries are symmetrical (it's because minY = -25, maxY = 25) image

Scenario 2: Expected behaviour in Y-direction

If we give a different values to minY, and maxY such that minY = 0 and maxY = 50 

# Front side (of vehicle) Point Cloud boundary for BEV
boundary = {
    "minX": 0,
    "maxX": 50,
    "minY": 0,
    "maxY": 50,
    "minZ": -2.73,
    "maxZ": 1.27
}

We can see that left and right boundary now starts at the left of the car image

Scenario 3: Unexpected behaviour in X-direction

However, any attempt to set minX < 0 will not result in the view at the back of the car. Here we set minX = -25 and maxX = 25

# Front side (of vehicle) Point Cloud boundary for BEV
boundary = {
    "minX": -25,
    "maxX": 25,
    "minY": -25,
    "maxY": 25,
    "minZ": -2.73,
    "maxZ": 1.27
}

We see unexpected behaviour, the view doesn't include the car in the middle of the image, and the labels are not in the correct position) image

How?

I added discretization options to src/config/kitti_config.py like here: https://github.com/maudzung/Complex-YOLOv4-Pytorch/blob/43a046cbbd25d24dacfae9ab8f1e7fd5f6d16ce9/src/config/kitti_config.py#L36-L37

Update all related function call to take in Discretization_X and Discretization_Y instead of just Discretization

The main changes come from src/data_process/kitti_bev_utils.py where I made a more generalized conversion from 3D point cloud to BEV image. https://github.com/maudzung/Complex-YOLOv4-Pytorch/blob/43a046cbbd25d24dacfae9ab8f1e7fd5f6d16ce9/src/data_process/kitti_bev_utils.py#L41-L48

Testing?

This PR produces expected behavior for scenario 3

Scenario 3: Unexpected behaviour in X-direction

# Front side (of vehicle) Point Cloud boundary for BEV
boundary = {
    "minX": -25,
    "maxX": 25,
    "minY": -25,
    "maxY": 25,
    "minZ": -2.73,
    "maxZ": 1.27
}

We see expected behaviour, the car is in the middle, all annotations are at the correct positions. (Note that KITTI dataset doesn't provide annotations for the cars at the back) image