Multiple video streams cannot visualize detection results at the same time

[YYLCyylc]

I used docker to deploy deepstream=6.2, and used four video streams as input for detection. The model used is yolov8s,batchsize=8, fp16. When the batchsize of streammux in config is set as 4/8, while the fps in 4 screens are all equal to 25, only one of the four screens in the launched rtsp stream will display the detection result at the same time. When batchsize is set to 1, the output rtsp stream seems good but there were errors like

[h264 @ 0000025197dff540] error while decoding MB 70 21
[h264 @ 000002519843fac0] Invalid level prefix
[h264 @ 000002519843fac0] error while decoding MB 57 35
[h264 @ 0000025197dff540] Invalid level prefix
[h264 @ 0000025197dff540] error while decoding MB 79 26

but when i try old version code using .wts model file , it works well and none of the above happened

[YYLCyylc]

this is the output of the problem

root@server58:/opt/nvidia/deepstream/deepstream-6.2/DeepStream-Yolo# deepstream-app -c deepstream_app_config.txt 

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8553/ds-test ***

0:00:03.276410420 12791 0x5638a7fc5c10 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/opt/nvidia/deepstream/deepstream-6.2/DeepStream-Yolo/yolov8s.onnx_b8_gpu0_fp16.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:610 [FullDims Engine Info]: layers num: 2
0   INPUT  kFLOAT input           3x640x640       min: 1x3x640x640     opt: 8x3x640x640     Max: 8x3x640x640     
1   OUTPUT kFLOAT output          8400x6          min: 0               opt: 0               Max: 0               

0:00:03.306486592 12791 0x5638a7fc5c10 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /opt/nvidia/deepstream/deepstream-6.2/DeepStream-Yolo/yolov8s.onnx_b8_gpu0_fp16.engine
0:00:03.319127811 12791 0x5638a7fc5c10 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/opt/nvidia/deepstream/deepstream-6.2/DeepStream-Yolo/config_infer_primary_yoloV8.txt sucessfully

Runtime commands:
        h: Print this help
        q: Quit

        p: Pause
        r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:239>: Pipeline ready

** INFO: <bus_callback:225>: Pipeline running

** INFO: <bus_callback:225>: Pipeline running

**PERF:  FPS 0 (Avg)    FPS 1 (Avg)     FPS 2 (Avg)     FPS 3 (Avg)
**PERF:  28.73 (28.14)  29.23 (28.74)   28.87 (28.33)   29.15 (28.48)
**PERF:  24.56 (25.25)  24.91 (25.68)   24.92 (25.61)   24.89 (25.64)

(deepstream-app:12791): GLib-GObject-WARNING **: 08:49:17.333: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'
**PERF:  25.42 (25.32)  25.14 (25.37)   25.10 (25.34)   25.10 (25.35)
**PERF:  24.90 (25.22)  24.85 (25.19)   24.89 (25.23)   24.92 (25.24)
**PERF:  25.09 (25.17)  25.11 (25.19)   25.11 (25.18)   25.09 (25.18)
^C** ERROR: <_intr_handler:140>: User Interrupted.. 

Quitting
nvstreammux: Successfully handled EOS for source_id=0
nvstreammux: Successfully handled EOS for source_id=1
nvstreammux: Successfully handled EOS for source_id=2
nvstreammux: Successfully handled EOS for source_id=3
App run successful

this is the output of the old version

deepstream-app -c deepstream_app_config.txt 

 *** DeepStream: Launched RTSP Streaming at rtsp://localhost:8554/ds-test ***

Opening in BLOCKING MODE 
Deserialize yoloLayer plugin: yolo
0:00:06.604767479 23728     0x3d2e7950 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1900> [UID = 1]: deserialized trt engine from :/home/nvidia/jingtianran/DeepStream-Yolo/model_b8_gpu0_fp16.engine
INFO: [Implicit Engine Info]: layers num: 5
0   INPUT  kFLOAT data            3x640x640       
1   OUTPUT kFLOAT num_detections  1               
2   OUTPUT kFLOAT detection_boxes 8400x4          
3   OUTPUT kFLOAT detection_scores 8400            
4   OUTPUT kFLOAT detection_classes 8400            

0:00:06.626526876 23728     0x3d2e7950 INFO                 nvinfer gstnvinfer.cpp:638:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2004> [UID = 1]: Use deserialized engine model: /home/nvidia/jingtianran/DeepStream-Yolo/model_b8_gpu0_fp16.engine
0:00:06.640273750 23728     0x3d2e7950 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary_gie> [UID 1]: Load new model:/home/nvidia/jingtianran/DeepStream-Yolo/yolov8_config.txt sucessfully
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency
cb_sourcesetup set 100 latency

Runtime commands:
        h: Print this help
        q: Quit

        p: Pause
        r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

**PERF:  FPS 0 (Avg)    FPS 1 (Avg)     FPS 2 (Avg)     FPS 3 (Avg)
**PERF:  0.00 (0.00)    0.00 (0.00)     0.00 (0.00)     0.00 (0.00)
** INFO: <bus_callback:194>: Pipeline ready

** INFO: <bus_callback:180>: Pipeline running

Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 261 
Opening in BLOCKING MODE 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteBlockCreate : Block : BlockType = 261 
** INFO: <bus_callback:180>: Pipeline running

NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA_ENC: bBlitMode is set to TRUE 
**PERF:  15.29 (14.82)  15.57 (14.82)   15.57 (14.82)   15.29 (14.82)

(deepstream-app:23728): GLib-GObject-WARNING **: 16:21:00.849: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'
**PERF:  14.34 (14.59)  14.33 (14.81)   14.33 (14.81)   14.34 (14.59)
**PERF:  14.28 (14.52)  14.09 (14.52)   14.09 (14.52)   14.28 (14.52)

[marcoslucianops]

Sorry but I think I didn't understand the issue, can you explain better?

When batchsize is set to 1, the output rtsp stream seems good but there were errors like

If I'm not wrong, in the ffmpeg/ffmpeg, this error is normal to get. The output should work normally. Aren't you getting the output on the RTSP?

[YYLCyylc]

Yes i can get the output of RTSP, but when the error occurs, the output stream will have garbled pixels. And what I am puzzled is that in the case of exactly the same configuration, using the onnx model detection can not display the detected bbox in multiple videos at the same time. Which means when i have 4 same input, there are 4 screens in the output RTSP, but the bbox only display in one of them at the same time. The new code is deployed in V100 using docker and the old code is deployed in jetson xavier NX. I compared the two outputs and guess if there is a problem about blocking mode

[marcoslucianops]

The output stream will have garbled pixels

This is related to the latency on the pipeline.

using the onnx model detection can not display the detected bbox in multiple videos at the same time

I just tested and I can get the bboxes at same time in multiple videos (YOLOv8s exported to ONNX).

[YYLCyylc]

Can you provide the output of the terminal？I want to compare with mine. Thanks.

[marcoslucianops]

I'm using a GTX 1050, YOLOv8s in FP32 model, batch-size = 2

[YYLCyylc]

I noticed that your model has four layers and in my output there are only two layers. I regenerated the onnx model after git pull, and now it is successfully resolved. Thank you very much!

[marcoslucianops]

The old implementation was having issues in the output in DeepStream < 6.2. I updated to fix it.