search

dusty-nv / jetson-inference

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
https://developer.nvidia.com/embedded/twodaystoademo
MIT License
7.81k stars 2.98k forks source link

Use the detectnet command to detect the video, the result of the detect video is distorted #1671

Open bladetin opened 1 year ago

bladetin commented 1 year ago

Command: detectnet ./combine.avi The output: image Source video: image Complete output:

(rgzy5) root@rgzy:/rg/8fusion-cpp/gst-cv-video-src/build# detectnet ./combine.avi
[gstreamer] initialized gstreamer, version 1.16.3.0
[gstreamer] gstDecoder -- creating decoder for ./combine.avi
[gstreamer] gstDecoder -- discovered video resolution: 1818x452  (framerate 25.000000 Hz)
[gstreamer] gstDecoder -- discovered video caps:  image/jpeg, framerate=(fraction)25/1, width=(int)1818, height=(int)452
[gstreamer] gstDecoder -- pipeline string:
[gstreamer] filesrc location=./combine.avi ! avidemux ! queue ! jpegdec name=decoder ! video/x-raw ! appsink name=mysink
[video]  created gstDecoder from file:///rg/8fusion-cpp/gst-cv-video-src/build/./combine.avi
------------------------------------------------
gstDecoder video options:
------------------------------------------------
  -- URI: file:///rg/8fusion-cpp/gst-cv-video-src/build/./combine.avi
     - protocol:  file
     - location:  ./combine.avi
     - extension: avi
  -- deviceType: file
  -- ioType:     input
  -- codec:      MJPEG
  -- codecType:  cpu
  -- width:      1818
  -- height:     452
  -- frameRate:  25
  -- numBuffers: 4
  -- zeroCopy:   true
  -- flipMethod: none
  -- loop:       0
------------------------------------------------
[OpenGL] glDisplay -- X screen 0 resolution:  2560x1440
[OpenGL] glDisplay -- X window resolution:    2560x1440
[OpenGL] glDisplay -- display device initialized (2560x1440)
[video]  created glDisplay from display://0
------------------------------------------------
glDisplay video options:
------------------------------------------------
  -- URI: display://0
     - protocol:  display
     - location:  0
  -- deviceType: display
  -- ioType:     output
  -- width:      2560
  -- height:     1440
  -- frameRate:  0
  -- numBuffers: 4
  -- zeroCopy:   true
------------------------------------------------

detectNet -- loading detection network model from:
          -- model        networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff
          -- input_blob   'Input'
          -- output_blob  'NMS'
          -- output_count 'NMS_1'
          -- class_labels networks/SSD-Mobilenet-v2/ssd_coco_labels.txt
          -- threshold    0.500000
          -- batch_size   1

[TRT]    TensorRT version 8.5.2
[TRT]    loading NVIDIA plugins...
[TRT]    Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[TRT]    Registered plugin creator - ::BatchedNMS_TRT version 1
[TRT]    Registered plugin creator - ::BatchTilePlugin_TRT version 1
[TRT]    Registered plugin creator - ::Clip_TRT version 1
[TRT]    Registered plugin creator - ::CoordConvAC version 1
[TRT]    Registered plugin creator - ::CropAndResizeDynamic version 1
[TRT]    Registered plugin creator - ::CropAndResize version 1
[TRT]    Registered plugin creator - ::DecodeBbox3DPlugin version 1
[TRT]    Registered plugin creator - ::DetectionLayer_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[TRT]    Registered plugin creator - ::EfficientNMS_TRT version 1
[TRT]    Could not register plugin creator -  ::FlattenConcat_TRT version 1
[TRT]    Registered plugin creator - ::GenerateDetection_TRT version 1
[TRT]    Registered plugin creator - ::GridAnchor_TRT version 1
[TRT]    Registered plugin creator - ::GridAnchorRect_TRT version 1
[TRT]    Registered plugin creator - ::GroupNorm version 1
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 1
[TRT]    Registered plugin creator - ::InstanceNormalization_TRT version 2
[TRT]    Registered plugin creator - ::LayerNorm version 1
[TRT]    Registered plugin creator - ::LReLU_TRT version 1
[TRT]    Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[TRT]    Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[TRT]    Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[TRT]    Registered plugin creator - ::NMSDynamic_TRT version 1
[TRT]    Registered plugin creator - ::NMS_TRT version 1
[TRT]    Registered plugin creator - ::Normalize_TRT version 1
[TRT]    Registered plugin creator - ::PillarScatterPlugin version 1
[TRT]    Registered plugin creator - ::PriorBox_TRT version 1
[TRT]    Registered plugin creator - ::ProposalDynamic version 1
[TRT]    Registered plugin creator - ::ProposalLayer_TRT version 1
[TRT]    Registered plugin creator - ::Proposal version 1
[TRT]    Registered plugin creator - ::PyramidROIAlign_TRT version 1
[TRT]    Registered plugin creator - ::Region_TRT version 1
[TRT]    Registered plugin creator - ::Reorg_TRT version 1
[TRT]    Registered plugin creator - ::ResizeNearest_TRT version 1
[TRT]    Registered plugin creator - ::ROIAlign_TRT version 1
[TRT]    Registered plugin creator - ::RPROI_TRT version 1
[TRT]    Registered plugin creator - ::ScatterND version 1
[TRT]    Registered plugin creator - ::SeqLen2Spatial version 1
[TRT]    Registered plugin creator - ::SpecialSlice_TRT version 1
[TRT]    Registered plugin creator - ::SplitGeLU version 1
[TRT]    Registered plugin creator - ::Split version 1
[TRT]    Registered plugin creator - ::VoxelGeneratorPlugin version 1
[TRT]    detected model format - UFF  (extension '.uff')
[TRT]    desired precision specified for GPU: FASTEST
[TRT]    requested fasted precision for device GPU without providing valid calibrator, disabling INT8
[TRT]    [MemUsageChange] Init CUDA: CPU +221, GPU +0, now: CPU 258, GPU 4196 (MiB)
[TRT]    Trying to load shared library libnvinfer_builder_resource.so.8.5.2
[TRT]    Loaded shared library libnvinfer_builder_resource.so.8.5.2
[TRT]    [MemUsageChange] Init builder kernel library: CPU +302, GPU +285, now: CPU 583, GPU 4502 (MiB)
[TRT]    native precisions detected for GPU:  FP32, FP16, INT8
[TRT]    selecting fastest native precision for GPU:  FP16
[TRT]    found engine cache file /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.1.1.8502.GPU.FP16.engine
[TRT]    found model checksum /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.sha256sum
[TRT]    echo "$(cat /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.sha256sum) /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff" | sha256sum --check --status
[TRT]    model matched checksum /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.sha256sum
[TRT]    loading network plan from engine cache... /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff.1.1.8502.GPU.FP16.engine
[TRT]    device GPU, loaded /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff
[TRT]    Loaded engine size: 33 MiB
[TRT]    Trying to load shared library libcublas.so.11
[TRT]    Loaded shared library libcublas.so.11
[TRT]    Using cublas as plugin tactic source
[TRT]    Trying to load shared library libcublasLt.so.11
[TRT]    Loaded shared library libcublasLt.so.11
[TRT]    Using cublasLt as core library tactic source
[TRT]    [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +534, GPU +379, now: CPU 871, GPU 4917 (MiB)
[TRT]    Trying to load shared library libcudnn.so.8
[TRT]    Loaded shared library libcudnn.so.8
[TRT]    Using cuDNN as plugin tactic source
[TRT]    Using cuDNN as core library tactic source
[TRT]    [MemUsageChange] Init cuDNN: CPU +83, GPU +42, now: CPU 954, GPU 4959 (MiB)
[TRT]    Deserialization required 1215333 microseconds.
[TRT]    [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +32, now: CPU 0, GPU 32 (MiB)
[TRT]    Trying to load shared library libcublas.so.11
[TRT]    Loaded shared library libcublas.so.11
[TRT]    Using cublas as plugin tactic source
[TRT]    Trying to load shared library libcublasLt.so.11
[TRT]    Loaded shared library libcublasLt.so.11
[TRT]    Using cublasLt as core library tactic source
[TRT]    [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 954, GPU 4959 (MiB)
[TRT]    Trying to load shared library libcudnn.so.8
[TRT]    Loaded shared library libcudnn.so.8
[TRT]    Using cuDNN as plugin tactic source
[TRT]    Using cuDNN as core library tactic source
[TRT]    [MemUsageChange] Init cuDNN: CPU +0, GPU +0, now: CPU 954, GPU 4959 (MiB)
[TRT]    Total per-runner device persistent memory is 9216
[TRT]    Total per-runner host persistent memory is 159632
[TRT]    Allocated activation device memory of size 7000064
[TRT]    [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +7, now: CPU 0, GPU 39 (MiB)
[TRT]    
[TRT]    CUDA engine context initialized on device GPU:
[TRT]       -- layers       106
[TRT]       -- maxBatchSize 1
[TRT]       -- deviceMemory 7000064
[TRT]       -- bindings     3
[TRT]       binding 0
                -- index   0
                -- name    'Input'
                -- type    FP32
                -- in/out  INPUT
                -- # dims  3
                -- dim #0  3
                -- dim #1  300
                -- dim #2  300
[TRT]       binding 1
                -- index   1
                -- name    'NMS'
                -- type    FP32
                -- in/out  OUTPUT
                -- # dims  3
                -- dim #0  1
                -- dim #1  100
                -- dim #2  7
[TRT]       binding 2
                -- index   2
                -- name    'NMS_1'
                -- type    FP32
                -- in/out  OUTPUT
                -- # dims  3
                -- dim #0  1
                -- dim #1  1
                -- dim #2  1
[TRT]    
[TRT]    binding to input 0 Input  binding index:  0
[TRT]    binding to input 0 Input  dims (b=1 c=3 h=300 w=300) size=1080000
[TRT]    binding to output 0 NMS  binding index:  1
[TRT]    binding to output 0 NMS  dims (b=1 c=1 h=100 w=7) size=2800
[TRT]    binding to output 1 NMS_1  binding index:  2
[TRT]    binding to output 1 NMS_1  dims (b=1 c=1 h=1 w=1) size=4
[TRT]    
[TRT]    device GPU, /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff initialized.
[TRT]    W = 7  H = 100  C = 1
[TRT]    detectNet -- maximum bounding boxes:   100
[TRT]    loaded 91 class labels
[TRT]    detectNet -- number of object classes:  91
[TRT]    loaded 0 class colors
[TRT]    didn't load expected number of class colors  (0 of 91)
[TRT]    filling in remaining 91 class colors with default colors
[gstreamer] opening gstDecoder for streaming, transitioning pipeline to GST_STATE_PLAYING
[gstreamer] gstreamer changed state from NULL to READY ==> mysink
[gstreamer] gstreamer changed state from NULL to READY ==> capsfilter0
[gstreamer] gstreamer changed state from NULL to READY ==> decoder
[gstreamer] gstreamer changed state from NULL to READY ==> queue0
[gstreamer] gstreamer changed state from NULL to READY ==> avidemux1
[gstreamer] gstreamer changed state from NULL to READY ==> filesrc0
[gstreamer] gstreamer changed state from NULL to READY ==> pipeline0
[gstreamer] gstreamer changed state from READY to PAUSED ==> capsfilter0
[gstreamer] gstreamer changed state from READY to PAUSED ==> decoder
[gstreamer] gstreamer stream status CREATE ==> src
[gstreamer] gstreamer changed state from READY to PAUSED ==> queue0
[gstreamer] gstreamer stream status ENTER ==> src
[gstreamer] gstreamer stream status CREATE ==> sink
[gstreamer] gstreamer changed state from READY to PAUSED ==> avidemux1
[gstreamer] gstreamer changed state from READY to PAUSED ==> filesrc0
[gstreamer] gstreamer stream status ENTER ==> sink
[gstreamer] gstDecoder -- onPreroll()
[gstreamer] gstreamer message stream-start ==> pipeline0
[gstreamer] gstreamer mysink taglist, video-codec=(string)"Motion\ JPEG", bitrate=(uint)7547532;
[gstreamer] gstreamer mysink taglist, encoder=(string)Lavf58.29.100, container-format=(string)AVI;
[gstreamer] gstBufferManager -- map buffer size was less than max size (1234864 vs 1234871)
[gstreamer] gstBufferManager recieve caps:  video/x-raw, format=(string)I420, width=(int)1818, height=(int)452, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)1/1, chroma-site=(string)jpeg, colorimetry=(string)1:4:0:0, framerate=(fraction)25/1
[gstreamer] gstBufferManager -- recieved first frame, codec=MJPEG format=i420 width=1818 height=452 size=1234871
[cuda]   allocated 4 ring buffers (1234871 bytes each, 4939484 bytes total)
[cuda]   allocated 4 ring buffers (8 bytes each, 32 bytes total)
[gstreamer] gstreamer changed state from READY to PAUSED ==> mysink
[gstreamer] gstreamer changed state from READY to PAUSED ==> pipeline0
[gstreamer] gstreamer message async-done ==> pipeline0
[gstreamer] gstreamer message new-clock ==> pipeline0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> mysink
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> capsfilter0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> decoder
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> queue0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> avidemux1
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> filesrc0
[gstreamer] gstreamer changed state from PAUSED to PLAYING ==> pipeline0
[cuda]   allocated 4 ring buffers (2465208 bytes each, 9860832 bytes total)
[OpenGL] glDisplay -- set the window size to 1818x452
[OpenGL] creating 1818x452 texture (GL_RGB8 format, 2465208 bytes)
[cuda]   registered openGL texture for interop access (1818x452, GL_RGB8, 2465208 bytes)

[TRT]    ------------------------------------------------
[TRT]    Timing Report /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff
[TRT]    ------------------------------------------------
[TRT]    Pre-Process   CPU   0.04451ms  CUDA   0.08986ms
[TRT]    Network       CPU   7.50483ms  CUDA   7.01840ms
[TRT]    Post-Process  CPU   0.01373ms  CUDA   0.05536ms
[TRT]    Total         CPU   7.56307ms  CUDA   7.16362ms
[TRT]    ------------------------------------------------

[TRT]    note -- when processing a single image, run 'sudo jetson_clocks' before
                to disable DVFS for more accurate profiling/timing measurements

1 objects detected

detected obj 0  class #5 (airplane)  confidence=0.747070
bounding box 0  (12.43, 0.33)  (1738.11, 451.00)  w=1725.68  h=450.67

[TRT]    ------------------------------------------------
[TRT]    Timing Report /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff
[TRT]    ------------------------------------------------
[TRT]    Pre-Process   CPU   0.02781ms  CUDA   0.09504ms
[TRT]    Network       CPU  11.44885ms  CUDA   7.59594ms
[TRT]    Post-Process  CPU   0.03040ms  CUDA   0.00227ms
[TRT]    Visualize     CPU   3.08336ms  CUDA   3.08707ms
[TRT]    Total         CPU  14.59042ms  CUDA  10.78032ms
[TRT]    ------------------------------------------------

1 objects detected

detected obj 0  class #5 (airplane)  confidence=0.610840
bounding box 0  (10.65, 0.66)  (1725.68, 451.78)  w=1715.03  h=451.12

[TRT]    ------------------------------------------------
[TRT]    Timing Report /usr/local/bin/networks/SSD-Mobilenet-v2/ssd_mobilenet_v2_coco.uff
[TRT]    ------------------------------------------------
[TRT]    Pre-Process   CPU   0.02509ms  CUDA   0.08838ms
[TRT]    Network       CPU   8.83468ms  CUDA   6.72522ms
[TRT]    Post-Process  CPU   0.01286ms  CUDA   0.01312ms
[TRT]    Visualize     CPU   0.05565ms  CUDA   0.63891ms
[TRT]    Total         CPU   8.92828ms  CUDA   7.46563ms
[TRT]    ------------------------------------------------
dusty-nv commented 1 year ago

@bladetin does video-viewer or video-viewer.py display your video properly?

If not, I think it may have to do with it being an uncommon resolution (1818x452) and MJPEG. Could you try a different video or resizing it with a different tool, so see if it has an impact?

Also, can you try video-viewer --headless combine.avi combine.mp4 and see if it transcodes the video properly? If so, the distortion is present in the OpenGL visualization and not the actual processing.