Major Bugs

[tjtanaa]

Hi, I have found some bugs in the implementations that you have. The corrected version of code could be found in my forked repository https://github.com/tjtanaa/PointPillars.git#5824e2d0c71f3d023d2cfa793fa4cced187cfafb.

In the processors.py

1. The transformation operations is wrong. The following is the correction.

   @staticmethod
   def transform_labels_into_lidar_coordinates(labels: List[Label3D], R: np.ndarray, t: np.ndarray):
       transformed = []
       for label in labels:
           label.centroid = (label.centroid - t) @ np.linalg.inv(R).T       # corrected 
           label.dimension = label.dimension[[2, 1, 0]] # h w l => l ,w ,h
           label.yaw = -label.yaw            # the label.yaw is in the camera frame, in the lidar frame, the yaw is negative of label.yaw
           label.yaw -= np.pi / 2
       ...

   Note: if you want to get the correct z-center coordinate of the bounding box, you have add an offset. In lidar frame:

   label.centroid[2] += label.dimension[2] / 2   # the height

In /src/point_pillars.cpp

1. The index for the bestAnchor allocation (in the end part of the code where there is no anchor with IoU > positive threshold) is wrong. The way to correct it is to store the bestAnchorXId and bestAnchorYId during the first for loop, when searching maxIoU between label box and the anchor. The corrected version can be found in my forked repository /src/point_pillars_v2.cpp at commit https://github.com/tjtanaa/PointPillars.git#5824e2d0c71f3d023d2cfa793fa4cced187cfafb

std::tuple<pybind11::array_t<float>, int, int> createPillarsTarget(const pybind11::array_t<float>& objectPositions,
                                             const pybind11::array_t<float>& objectDimensions,
                                             const pybind11::array_t<float>& objectYaws,
                                             const pybind11::array_t<int>& objectClassIds,
                                             const pybind11::array_t<float>& anchorDimensions,
                                             const pybind11::array_t<float>& anchorZHeights,
                                             const pybind11::array_t<float>& anchorYaws,
                                             float positiveThreshold,
                                             float negativeThreshold,
                                             int nbClasses,
                                             int downscalingFactor,
                                             float xStep,
                                             float yStep,
                                             float xMin,
                                             float xMax,
                                             float yMin,
                                             float yMax,
                                             float zMin,
                                             float zMax,
                                             bool printTime = false)
{
       ...
        float maxIou = 0;
        BoundingBox3D bestAnchor = {};
        int bestAnchorId = 0;
        int bestAnchorXId = 0; # add variable
        int bestAnchorYId = 0; # add variable

        ...

                    if (maxIou < iouOverlap)
                    {
                        maxIou = iouOverlap;
                        bestAnchor = anchorBox;
                        bestAnchorId = anchorCount;
                        bestAnchorXId = xId;
                        bestAnchorYId = yId;
                    }
       ...
     if (maxIou < positiveThreshold) // Comparing maxIOU for that object obtained after checking with every anchor box
            // If none of the anchors passed the threshold, then we place the best anchor details for that object. 
        {
            negCnt++;

            const auto xId = bestAnchorXId;
            const auto yId = bestAnchorYId;
            ...
      }
       ...

2. I am aware that you have followed the Point-Pillar Papers. However, the paper states that it uses the same loss as SECOND. I believe the angle loss in the Point-Pillar is wrong and the one in the SECOND paper is the correct version. I have referred to the original official code from https://github.com/traveller59/second.pytorch.git , I am confident that the loss in Point-Pillar Papers for angle regression is wrong.

Please refer to files from the official repository (particularly: second/core/box_np_ops.py and second/pytorch/models/voxelnet.py)

In second/core/box_np_ops.py, functions second_box_decode and second_box_encode. In second/pytorch/models/voxelnet.py, functions add_sin_difference.

Thus, when preparing the targets in the function createPillarsTarget, the angle target should be

            // tensor.mutable_at(objectCount, xId, yId, bestAnchorId, 7) = std::sin(labelBox.yaw - bestAnchor.yaw); # old version
            tensor.mutable_at(objectCount, xId, yId, bestAnchorId, 7) = labelBox.yaw - bestAnchor.yaw; # corrected version

Based on the formula for angle loss in SECOND, the sine function is applied to

   Let target_delta_angle = tensor.mutable_at(objectCount, xId, yId, bestAnchorId, 7) = labelBox.yaw - bestAnchor.yaw;
   Let predicted_delta_angle = tensor_infered_by_point_pillar_model
   loss = tf.math.sin(target_delta_angle - predicted_delta_angle) # correct (in SECOND paper)
   # loss = tf.math.sin( target_angle - predicted_angle) # wrong (in Point Pillar paper)

_If you adopted the above changes the bb_yaw in the inference_utils.py should be bb_yaw = ang[value] + real_anchors[i][4] instead._ For reference, please refer to inference_utils_v2.py in my forked repository.

[tyagi-iiitv]

Hi @tjtanaa, it would be great if you could contribute to this repo directly. Since I'm not actively contributing to this project now, that'll help in continued development and improvement. Thanks for your consideration.

[Mfertega]

Hi @tjtanaa, I am trying to deploy your solution because it is more accurate with the dimensions use in the paper but I am having trouble to understand where do you get the det3d library, I guess is a different solution equivalent for the datareader, but could you please send me the library so I can deploy it?

[tjtanaa]

Hi @Mfertega please clone and install the following repository https://github.com/tjtanaa/det3d.git . Let me know if you have any problem setting up the package

[ma7555]

Hello @tjtanaa, Your repo does not compile.

CMake Error at pybind11/tools/pybind11Tools.cmake:151 (add_library):
  Cannot find source file:

    src/point_pillars_v2.cpp

Please allow issues to be opened on your repo. Thanks!

[tjtanaa]

Hello @tjtanaa, Your repo does not compile.

CMake Error at pybind11/tools/pybind11Tools.cmake:151 (add_library):
  Cannot find source file:

    src/point_pillars_v2.cpp

Please allow issues to be opened on your repo. Thanks!

Hi @ma7555 for now you could comment out the pybind11_add_module(point_pillars_v2 SHARED src/point_pillars_v2.cpp) from the CMakeLists.txt. The should solve your compilation problem.

[Manueljohnson063]

@tjtanaa Hi sir how to feed the data to visualise tht output bounding box