Incorrect Keypoint and Bounding Box Outputs with RetinaFace Custom Parser in DeepStream 6.3

[sowmiya-masterworks]

I'm using the RetinaFace custom parser from the face-recognition-deepstream repo and encountering several issues concerning the output on keypoint detection and bounding box accuracy, using DeepStream 6.3.

Environment Hardware: NVIDIA GeForce RTX 3060 Driver Version: 555.42.06 CUDA Version: 12.5 DeepStream Version: 6.3 Operating System: [Please specify your OS, e.g., Ubuntu 20.04] Test Applications: DeepStream Test5 (both C++ and Python versions) Models and Weights Models Tried: Both ResNet50 and MobileNet architectures were tested. Weights: The model weights used are sourced from the [Pytorch_Retinaface repository]https://github.com/biubug6/Pytorch_Retinaface Expected Behavior Accurate detection and output of face landmarks and bounding boxes in the video stream.

Actual Behavior Landmarks: Many keypoint coordinates are either zero or negative, which does not correspond to valid pixel values. Bounding Boxes: Outputs are often unrealistic (e.g., exceedingly large dimensions). Video Output: No detections appear in the output video. Steps to Reproduce Tested the setup using the DeepStream Test5 application for both primary inference engine (pgie) and secondary inference engine (sgie). Also tested using the Python3 main.py script provided in the repository. In all tests, inappropriate bounding boxes and keypoints were observed across different setups and models. Console OutputRaw output array: output[0] = 0.937012 output[1] = 1.41797 output[2] = -1.57422 output[3] = -0.820801 output[4] = 1.58301 output[5] = 0.605469 output[6] = -0.943848 output[7] = -0.15332 output[8] = -1.27832 output[9] = 1.35547 output[10] = -0.252197 output[11] = -0.817383 output[12] = 0.0300903 output[13] = 1.14355 output[14] = -0.243774 output[15] = -0.335449 Raw output array: output[0] = 0.937012 output[1] = 1.41797 output[2] = -1.57422 output[3] = -0.820801 output[4] = 1.58301 output[5] = 0.605469 output[6] = -0.943848 output[7] = -0.15332 output[8] = -1.27832 output[9] = 1.35547 output[10] = -0.252197 output[11] = -0.817383 output[12] = 0.0300903 output[13] = 1.14355 output[14] = -0.243774 output[15] = -0.335449 Clipped BBox: 1.41797, 0, 0, 1.58301 Detection: Top: 0 Left: 1 Width: 4.29497e+09 Height: 1 Confidence: 0.605469 Landmarks: 0 0 -1 1 0 0 0 1 0 0 Raw output array: output[0] = 0.935547 output[1] = 1.41797 output[2] = -1.57324 output[3] = -0.819336 output[4] = 1.58203 output[5] = 0.60498 output[6] = -0.943359 output[7] = -0.15332 output[8] = -1.27734 output[9] = 1.35352 output[10] = -0.25293 output[11] = -0.816895 output[12] = 0.0317383 output[13] = 1.14355 output[14] = -0.243042 output[15] = -0.334473 Raw output array: output[0] = 0.935547 output[1] = 1.41797 output[2] = -1.57324 output[3] = -0.819336 output[4] = 1.58203 output[5] = 0.60498 output[6] = -0.943359 output[7] = -0.15332 output[8] = -1.27734 output[9] = 1.35352 output[10] = -0.25293 output[11] = -0.816895 output[12] = 0.0317383 output[13] = 1.14355 output[14] = -0.243042 output[15] = -0.334473 Clipped BBox: 1.41797, 0, 0, 1.58203 Detection: Top: 0 Left: 1 Width: 4.29497e+09 Height: 1 Confidence: 0.60498 Landmarks: 0 0 -1 1 0 0 0 1 0 0

[Athuliva]

with retinaface resnet50 i am getting correct bounding box. can you tell me how u generated engine file from https://github.com/biubug6/Pytorch_Retinaface?

[sowmiya-masterworks]

@Athuliva https://github.com/biubug6/Pytorch_Retinaface/blob/master/convert_to_onnx.py this script for onxx converstion and for engine conversion: /usr/src/tensorrt/bin/trtexec --onnx=FaceDetector.onnx --explicitBatch --workspace=204 --saveEngine=FaceDetector.engine --fp16 in docker deepstream 6.3

[sowmiya-masterworks]

@Athuliva https://github.com/wang-xinyu/tensorrtx/tree/master/retinaface while using this way generated engine file, i was facing error when using with the deep stream test5 application!

[Athuliva]

@sowmiya-masterworks have u loaded the libdecodeplugin.so while using https://github.com/wang-xinyu/tensorrtx/tree/master/retinaface

import ctypes
ctypes.cdll.LoadLibrary('/VA/retinaface_r50_63/R50/libdecodeplugin.so')

[zhouyuchong]

@sowmiya-masterworks try this, BTW, how do you decode those bbox and lmks? Since Retinaface is an anchor based model, the raw outputs should be post-processed otherwise they are unreadable.

[sowmiya-masterworks]

@zhouyuchong, thanks for the suggestion! Could you provide some guidance or recommend a repository for decoding the bounding boxes and landmarks from RetinaFace inside the DeepStream environment? Since it's an anchor-based model, I understand that the raw outputs need post-processing to be interpretable, and any pointers on how to approach this within DeepStream would be greatly appreciated.

[zhouyuchong]

@sowmiya-masterworks cpp version if you use custom-lib-path in nvinfer config path, note there is no support for landmarks in official datastructure. python version post-process. if you want to apply it to deepstream, just get raw outputs which I think you already knew, then do post-process in gst-probe callback function.