Any plans to officially support the 'NonMaxSuppression' operator from ONNX?

fengyuentau commented 3 years ago

System information (version)

OpenCV => 4.5.1
Operating System / Platform => Ubuntu 18.04/20.04
Compiler => gcc 7.5.0 for Ubuntu 18.04 / gcc 9.3.0 for Ubuntu 20.04

Detailed description

ONNX has a built-in node for NonMaxSuppression since version>=10, and PyTorch also has torchvision.ops.nms since version>=1.2. What I want to say is that it is very natural for PyTorch models to use torchvision.ops.nms in their forward test stage if they want to put every operation in a single model (ONNX). And of course, PyTorch has already supported to export an ONNX model with torchvision.ops.nms. However, I both checked OpenCV's wiki and the onnx_importer.cpp, neither of them showed the support for the 'NonMaxSuppression' op in the current version of OpenCV DNN.

I also learned from https://github.com/amdegroot/ssd.pytorch/pull/462 that the NMS is implemented inside the 'DetectionOutput' node, which is a custom operator from SSD. However, this is not adequate since 'DetectionOutput' has strict requirements with the input. What about a multi-tasking model (face detection + landmark detection)? The node can only work with bboxes but not the landmarks. And also the example to build the node from PyTorch is too complicated without clear documentations, and it seems to only work with a specific version of PyTorch. At least I failed with the later version of PyTorch (1.7.0).

An implementation of 'NonMaxSuppression' node can solve all the above questions. So, any plans to officially support the 'NonMaxSuppression' operator from ONNX? I can make a PR for this if this is not coming in near future.

Steps to reproduce

Any ONNX models with a 'NonMaxSuppression' node. I can share my ONNX model if you don't have one.

Issue submission checklist

[ ] I report the issue, it's not a question
[x] I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found solution
[x] I updated to latest OpenCV version and the issue is still there
[x] There is reproducer code and related data files: videos, images, onnx, etc

rogday commented 3 years ago

Hi, @fengyuentau, can you provide a way to generate NonMaxSuppression layer using PyTorch? Do you still have plans to implement this layer yourself?

fengyuentau commented 3 years ago

@rogday It is half-way done. I was working on opencv zoo. Hopefully I will figure it out in the following week. Here is an example to generate NonMaxSuppression layer using torchvision: https://github.com/fengyuentau/opencv_extra/blob/f3ca5a1acda5fc51c7be6eabc003487d781254b7/testdata/dnn/onnx/generate_onnx_models.py#L1499-L1515

opencv / opencv