3D Non Max Suppression feature request

[gdavid57]

Following the first thread in tensorflow/tensorflow.

Describe the feature and the current behavior/state.

TensorFlow already incorporates a native Non Max Suppression algorithm for 2D bounding boxes. It is used for instance for objet detection tasks, usually after a anchor box generation step as it is observed in YOLO. Another remarkable use exists within the Mask R-CNN algorithm: many anchor boxes are generated (~200,000-300,000 boxes). After a step to reduce this number to 6,000 using some classification results, tf.image.non_max_suppression is called to sort these 6,0000 best anchors and finally obtain 2,000 regions of interest.

The feature would consist of a Non Max Suppression algorithm for three dimensional bounding boxes. This 3D Non Max Suppression would almost have the same parameters than the existing 2D implementation, called with

tf.image.non_max_suppression(
    boxes, scores, max_output_size, iou_threshold=0.5,
    score_threshold=float('-inf'), name=None
)

with boxes containing no more the 4 coordinates of the two points that define a 2D bounding box, but the 6 coordinates of the two points defining a 3D bounding box. The other parameters would stay the same. The heart of this addition would be to adapt the IOU (Intersection Over Union) function within the Non Max Suppression C file to a three dimensional IOU evaluation.

Relevant information

• Are you willing to contribute it (yes/no): Yes although I've almost no experience in C++ If you wish to contribute, then read the requirements for new contributions in CONTRIBUTING.md
• Are you willing to maintain it going forward? (yes/no): Yes but it will not be necessary
• Is there a relevant academic paper? (if so, where): No
• Does the relavent academic paper exceed 50 citations? (yes/no): No
• Is there already an implementation in another framework? (if so, where): Some tried to implement it in PyTorch (see benefice part)
• Was it part of tf.contrib? (if so, where): No

Which API type would this fall under (layer, metric, optimizer, etc.)

It falls under the image processing API part (tf.image). The feature would add a Non Max Suppression algorithm _tf.image.non_max_suppression3d to TensorFlow for 3D bounding boxes following the structure of the existing 2D implementation. It will add three files to tensorflow/tensorflow/core/kernels/image/ : _non_max_suppression_3dop.cc, _non_max_suppression_3dop.cu.cc and _non_max_suppression_3dop.h.

Who will benefit with this feature?

This addon will greatly benefit to the increasing community of 3D object detection or 3D instance segmentation. The demand is increasing. I give two examples: this stackoverflow thread (alas removed by its author now), and this implementation in PyTorch that possesses a not working 3D implementation of the NMS (the implementation author have been asked many times to correct it). It actually exists some implementations in Python language but the speed performance given by a C implementation would be crucial when handling 3D data.

Any other info.

[bhack]

Can you check the function we have in Tensorflow 3d?

Probably they could be interested to have a high performance implementation of this op.

It was presented one months ago: https://ai.googleblog.com/2021/02/3d-scene-understanding-with-tensorflow.html

[gdavid57]

@bhack I've implemented 3D versions of Non Max Suppression and Crop And Resize algorithms in C langage, compile them within tensorflow sources, run basic tests with them, and used them successfully within a deep network. Before following the Contributing guidelines, I believe it is still up to you to decide if these algorithms should be included in tensorflow-addons as this thread is still marked as "ecosystem-review". Do you think it is relevant for tensorflow/addons?

[bhack]

I am monitoring your last comment at https://github.com/google-research/google-research/issues/660

[bhack]

It could be interesting to have a learnable 3d nms like:

https://github.com/abhi1kumar/groomed_nms/blob/main/lib/groomed_nms.py

[seanpmorgan]

TensorFlow Addons is transitioning to a minimal maintenance and release mode. New features will not be added to this repository. For more information, please see our public messaging on this decision: TensorFlow Addons Wind Down

Please consider sending feature requests / contributions to other repositories in the TF community with a similar charters to TFA: Keras Keras-CV Keras-NLP

[gdavid57]

3D Non Max Suppression and 3D Crop And Resize: https://github.com/gdavid57/3d-nms-car-custom-op