All the objects detected by retinafce is not in the deepstream meta

[Athuliva]

I added this line to check the retinaface detections

        if(r.score<=VIS_THRESH) continue;
        std::cout << r.score << "\n";

and also a probe to the primary detector and print the bounding box info. some objects in the metafile is missing when compared with the print statement in .so file

  def dummy_probe(self, pad, info, u_data):

        gst_buffer = info.get_buffer()
        if not gst_buffer:
            logger.error("Unable to get GstBuffer ")
            return

        batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
        if not batch_meta:
            return Gst.PadProbeReturn.OK

        l_frame = batch_meta.frame_meta_list
        while l_frame is not None:
            try:
                frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            except StopIteration:
                continue
            try:
                source_id = frame_meta.source_id
                payload = {}
                payload['objects'] = []
                l_obj = frame_meta.obj_meta_list
                while l_obj is not None:
                    try:
                        obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
                    except StopIteration:
                        continue

                    object_ = {}
                    object_["y"] = obj_meta.rect_params.top / self.frame_height
                    object_['x'] = obj_meta.rect_params.left / self.frame_width
                    object_['w'] = obj_meta.rect_params.width / self.frame_width
                    object_['h'] = obj_meta.rect_params.height / self.frame_height
                    object_['class_id'] = self.labels[obj_meta.class_id]
                    object_['trackingId'] = obj_meta.object_id
                    object_['confidence'] = obj_meta.confidence
                    logger.debug(object_)
                    payload['objects'].append(object_)

                    try:
                        l_obj = l_obj.next
                    except StopIteration:
                        break
                print('\n'.join([str(o['confidence']) for o in payload['objects']]))

            except Exception as e:
                logger.error(str(e))
            try:
                l_frame = l_frame.next
            except StopIteration:
                break

        return Gst.PadProbeReturn.OK

[zhouyuchong]

I added this line to check the retinaface detections

        if(r.score<=VIS_THRESH) continue;
        std::cout << r.score << "\n";

and also a probe to the primary detector and print the bounding box info. some objects in the metafile is missing when compared with the print statement in .so file

  def dummy_probe(self, pad, info, u_data):

        gst_buffer = info.get_buffer()
        if not gst_buffer:
            logger.error("Unable to get GstBuffer ")
            return

        batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
        if not batch_meta:
            return Gst.PadProbeReturn.OK

        l_frame = batch_meta.frame_meta_list
        while l_frame is not None:
            try:
                frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
            except StopIteration:
                continue
            try:
                source_id = frame_meta.source_id
                payload = {}
                payload['objects'] = []
                l_obj = frame_meta.obj_meta_list
                while l_obj is not None:
                    try:
                        obj_meta = pyds.NvDsObjectMeta.cast(l_obj.data)
                    except StopIteration:
                        continue

                    object_ = {}
                    object_["y"] = obj_meta.rect_params.top / self.frame_height
                    object_['x'] = obj_meta.rect_params.left / self.frame_width
                    object_['w'] = obj_meta.rect_params.width / self.frame_width
                    object_['h'] = obj_meta.rect_params.height / self.frame_height
                    object_['class_id'] = self.labels[obj_meta.class_id]
                    object_['trackingId'] = obj_meta.object_id
                    object_['confidence'] = obj_meta.confidence
                    logger.debug(object_)
                    payload['objects'].append(object_)

                    try:
                        l_obj = l_obj.next
                    except StopIteration:
                        break
                print('\n'.join([str(o['confidence']) for o in payload['objects']]))

            except Exception as e:
                logger.error(str(e))
            try:
                l_frame = l_frame.next
            except StopIteration:
                break

        return Gst.PadProbeReturn.OK

Nice work!

[Athuliva]

I realize some objects are missing because there is a filter to drop faces of small sizes (and the values I inserted were high) in the arcface config. how can i extract the landmarks from the NvDsObjectMeta

[zhouyuchong]

@Athuliva set output-tensor-meta to true, then you can extract data from original model output.

[Athuliva]

i saw this in the updated readme decode landmarks in object_user_metadata commit ID. Does that mean now we can access the landmarks from the user meta using python?

also i made a change in the retinaface face config after seeing changes in the commit output-blob-names=bbox;lmk;conf from output-blob-names=prob

now i get these errors in the log ERROR: [TRT]: 3: Cannot find binding of given name: bbox ERROR: [TRT]: 3: Cannot find binding of given name: lmk ERROR: [TRT]: 3: Cannot find binding of given name: conf

ERROR: [TRT]: 3: Cannot find binding of given name: bbox
0:00:05.995957645   589      0x307bc70 WARN                 nvinfer gstnvinfer.cpp:713:gst_nvinfer_logger:<primary-inference-sub-0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1867> [UID = 1]: Could not find output layer 'bbox' in engine
ERROR: [TRT]: 3: Cannot find binding of given name: lmk
0:00:05.997177963   589      0x307bc70 WARN                 nvinfer gstnvinfer.cpp:713:gst_nvinfer_logger:<primary-inference-sub-0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1867> [UID = 1]: Could not find output layer 'lmk' in engine
ERROR: [TRT]: 3: Cannot find binding of given name: conf
0:00:05.997203459   589      0x307bc70 WARN                 nvinfer gstnvinfer.cpp:713:gst_nvinfer_logger:<primary-inference-sub-0> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::checkBackendParams() <nvdsinfer_context_impl.cpp:1867> [UID = 1]: Could not find output layer 'conf' in engine

update : i was using retinanet50 from https://github.com/wang-xinyu/tensorrtx/tree/master/retinaface, the latest model can be found at https://drive.google.com/file/d/1edsy2rKpQXAylp56bx3yqgsqkCE3Hue0/view

how did you convert onnx file to engine file? using the deep stream application itself?

[zhouyuchong]

@Athuliva Q1: yes, since it is in usermeta_data now, I follow the official cpp example so it should work... Q2: i'm not using retinaface from tensorrtx anymore. However, they are almost the same, just some differences between post-process. you can use the old one still, all need to be done is to rewrite the postprocess codes in nvdsinfer_customparser

[Athuliva]

Q2. with this new model update ctypes.cdll.LoadLibrary('/opt/models/retinaface/libplugin_rface.so') is not needed?. Also the old models are working with the `nvdsinfer_customparser' but not the new model. retinaface do not show how we built the engine file, Do you have some reference?

[zhouyuchong]

@Athuliva Yes, you don't need that plugin anymore. trans onnx to trt is simple, you can find many examples online. there is a simple script according to official TensorRT.

'''
Author: zhouyuchong
Date: 2024-05-30 11:15:58
Description: 
LastEditors: zhouyuchong
LastEditTime: 2024-07-16 16:45:05
'''
import os
import argparse
from loguru import logger
import tensorrt as trt
import math

TRT_LOGGER = trt.Logger()

def build_engine(onnx_file_path, engine_file_path="", set_input_shape=None):
    """Takes an ONNX file and creates a TensorRT engine to run inference with"""
    network_creation_flag = 0
    network_creation_flag = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)

    with trt.Builder(TRT_LOGGER) as builder, builder.create_network(
        network_creation_flag
    ) as network, builder.create_builder_config() as config, trt.OnnxParser(
        network, TRT_LOGGER
    ) as parser, trt.Runtime(
        TRT_LOGGER
    ) as runtime:
        config.set_memory_pool_limit(
            trt.MemoryPoolType.WORKSPACE, 1 << 28
        )  # 256MiB
        # Parse model file
        if not os.path.exists(onnx_file_path):
            print(
                "ONNX file {} not found, please run yolov3_to_onnx.py first to generate it.".format(
                    onnx_file_path
                )
            )
            exit(0)
        print("Loading ONNX file from path {}...".format(onnx_file_path))
        with open(onnx_file_path, "rb") as model:
            print("Beginning ONNX file parsing")
            if not parser.parse(model.read()):
                print("ERROR: Failed to parse the ONNX file.")
                for error in range(parser.num_errors):
                    print(parser.get_error(error))
                return None
        # set input shape
        profile = builder.create_optimization_profile()
        logger.debug("total input layer: {}".format(network.num_inputs))
        logger.debug(network.num_outputs)
        output = network.get_output(0)
        logger.debug(output.shape)
        for i in range(network.num_inputs):
            input = network.get_input(i)
        #     assert input.shape[0] == -1
            logger.debug("input layer-{}: {}".format(i, input.name))
        profile.set_shape("images", set_input_shape[0], set_input_shape[1], set_input_shape[2])
        config.add_optimization_profile(profile)
        logger.debug("build, may take a while...")

        plan = builder.build_serialized_network(network, config)
        engine = runtime.deserialize_cuda_engine(plan)
        print("Completed creating Engine")
        with open(engine_file_path, "wb") as f:
            f.write(plan)
        return engine
def main(args):
    trt_file_name = args.onnx.replace('.onnx', '_bs{}.trt'.format(args.batch))
    size = [int(x) for x in args.size.split('x')]
    input_shape = [(1, 3, size[0], size[1]), (math.ceil(args.batch/2), 3, size[0], size[1]), (args.batch, 3, size[0], size[1])]
    logger.debug("set input shape: {}".format(input_shape))
    build_engine(args.onnx, trt_file_name, input_shape)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--onnx', type=str, default='', help='onnx path', required=True)
    parser.add_argument('-s', '--size', type=str, default="640x640", help='input shape')
    parser.add_argument('-b', '--batch', type=int, default=1, help='max batch size')
    args = parser.parse_args()
    main(args=args)

and you can use command as well, just follow the official tensorrt doc.

as for the custom parser for retinaface, there is one in repo.

I strongly suggest using official tensorrt instead of tensorrtx, since there are many new features and tensorrtx doesn't make adaption, some codes can't even work.

[Manuja1998]

Q1: yes, since it is in usermeta_data now, I follow the official cpp example so it should work...

how can i extract the landmarks from user meta? user_meta.landmarks or something like that?

[zhouyuchong]

@Manuja1998 cpp example, python should also work but I didn't try.

[Manuja1998]

type of the meta type is NvDSMetaType.???

if user_meta and user_meta.base_meta.meta_type == pyds.NvDsMetaType.NVDSINFER_TENSOR_OUTPUT_META:
     pass
else:
      print( user_meta.base_meta.meta_type)

[zhouyuchong]

@Manuja1998 NVDS_USER_OBJECT_META_EXAMPLE I guess.

[Manuja1998]

user_meta.base_meta.meta_type is NvDsMetaType.??? I think there i no pybinding to extract this into python

[Athuliva]

can you try casting it using ctype

[zhouyuchong]

@Manuja1998 yes your are right, seems there is no python binding for this type. As @Athuliva said, can you try using ctype?

[Athuliva]

I tried this

                        elif user_meta.base_meta.meta_type == pyds.nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"):
                            user_meta_data = ctypes.cast(user_meta.user_meta_data, ctypes.POINTER(ctypes.c_int))

i got wrong type error. @zhouyuchong may be can you give some insights on how to caste it to python

[Athuliva]

This worked for me

if user_meta.base_meta.meta_type == pyds.nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"):
        user_meta_data = user_meta.user_meta_data
        ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
        ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
        pointer = ctypes.pythonapi.PyCapsule_GetPointer(user_meta_data, None)
        pointer = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_int))
        for i in range(10):
             print(pointer[i])

[zhouyuchong]

This worked for me

if user_meta.base_meta.meta_type == pyds.nvds_get_user_meta_type("NVIDIA.NVINFER.USER_META"):
        user_meta_data = user_meta.user_meta_data
        ctypes.pythonapi.PyCapsule_GetPointer.restype = ctypes.c_void_p
        ctypes.pythonapi.PyCapsule_GetPointer.argtypes = [ctypes.py_object, ctypes.c_char_p]
        pointer = ctypes.pythonapi.PyCapsule_GetPointer(user_meta_data, None)
        pointer = ctypes.cast(pointer, ctypes.POINTER(ctypes.c_int))
        for i in range(10):
             print(pointer[i])

good job!!