search

insight-platform / Savant

Python Computer Vision & Video Analytics Framework With Batteries Included
https://savant-ai.io
Apache License 2.0
553 stars 45 forks source link

Problems with processing images of different sizes #783

Closed denisvmedyantsev closed 2 months ago

denisvmedyantsev commented 3 months ago

Prepare a set of images of different sizes. E.g. download COCO val http://images.cocodataset.org/zips/val2017.zip

I used a subset of COCO val, 749 images, simple pipeline image source adapter -> module -> image sink adapter module.tar.gz

Problems:

  1. Processing took a while. The start was speedy. Then the pipeline got slower and slower. The final log shows only 3.66 FPS
  2. Not all images were processed. There is a value frame_delta=265 in the logs. There are 503 lines in the output json, half of them are EndOfStream, so only 251 frames were output.
  3. Try setting output_frame: codec: jpeg - there will be a GPU memory leak (you can see it clearly in the nvtop graph).

Log

module-1        |  INFO  savant_core::pipeline::stats                       > Frame-based FPS counter triggered: FPS = 3.66, OPS = 0.00, frame_delta = 265, time_delta = 72.34 sec, period=[1718294383537, 1718294455877] ms
bwsw commented 3 months ago

Fix in 0.2.x, 0.4.x.

tomskikh commented 3 months ago

Looks like it takes 0.3s to create and initialize nvjpegdec

2024-06-27T06:26:34.375742065Z  DEBUG insight::savant::savant_rs_video_decode_bin                           > Added pad src_coco-images on element savant_rs_video_demux+savantrsvideodemux0
2024-06-27T06:26:34.375882629Z  INFO  insight::savant::savant_rs_video_decode_bin                           > Adding branch with source coco-images
2024-06-27T06:26:34.376072135Z  DEBUG insight::savant::savant_rs_video_decode_bin                           > Received message <flags GST_MESSAGE_STATE_CHANGED of type Gst.MessageType> from nvjpegdec1
2024-06-27T06:26:34.376277981Z  DEBUG insight::savant::savant_rs_video_decode_bin                           > Received message <flags GST_MESSAGE_STATE_CHANGED of type Gst.MessageType> from nvjpegdec1
2024-06-27T06:26:34.376908103Z Using GPU 0 (NVIDIA GeForce RTX 3060 Laptop GPU, 30 SMs, 1536 th/SM max, CC 8.6, ECC off)
2024-06-27T06:26:34.632570190Z  DEBUG insight::savant::savant_rs_video_decode_bin                           > Attaching pad decodebin-coco-images.src_0 to ghost pad src_coco-images
tomskikh commented 3 months ago

We don't need to send EOS every time resolution changes on JPEG input. At least for nvjpegdec on dGPU.

UPD: Savant can process JPEG or PNG input with images of different sizes without sending EOS after each image.

tomskikh commented 3 months ago

3. Try setting output_frame: codec: jpeg - there will be a GPU memory leak (you can see it clearly in the nvtop graph).

Opened a bug in https://forums.developer.nvidia.com/t/nvjpegenc-dont-release-gpu-memory-when-gst-element-removed-from-pipeline/298248

tomskikh commented 3 months ago

2. Not all images were processed. There is a value frame_delta=265 in the logs. There are 503 lines in the output json, half of them are EndOfStream, so only 251 frames were output.

That was because of timeout in sending message to ZMQ socket. Setting RECEIVE_TIMEOUT_MSECS=30000 in source adapter fixes it.

tomskikh commented 3 months ago

We don't need to send EOS every time resolution changes on JPEG input. At least for nvjpegdec on dGPU.

UPD: Savant can process JPEG or PNG input with images of different sizes without sending EOS after each image.

On Jetson this doesn't work for JPEG. I'll make a parameter to disable generating EOS when frame resolution changes in source adapter and module.

tomskikh commented 3 months ago
  1. Try setting output_frame: codec: jpeg - there will be a GPU memory leak (you can see it clearly in the nvtop graph).

Opened a bug in https://forums.developer.nvidia.com/t/nvjpegenc-dont-release-gpu-memory-when-gst-element-removed-from-pipeline/298248

Cannot reuse existing nvjpegenc on jetson as a workaround:

Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 277 
NvMMLiteBlockCreate : Block : BlockType = 277 
NVJPGGetSurfPinHandle 220: NVJPGGetSurfPinHandle : Surface not registered 
NVJPGPushSurfFalconMethodRelocShift 470: Surface not registered 
JPEGEncFeedFrame 1787: NVJPGPushSurfFalconMethodRelocShift failed for SET_CUR_PIC_CHROMA_U
NVJPGGetSurfPinHandle 220: NVJPGGetSurfPinHandle : Surface not registered 
NVJPGPushSurfFalconMethodRelocShift 470: Surface not registered 
JPEGEncFeedFrame 1794: NVJPGPushSurfFalconMethodRelocShift failed for SET_CUR_PIC_CHROMA_V
NVJPGPushStreamEnd 253: Stream end failed
JPEGEncFeedFrame 1800: Call to pushNVJPGStreamEnd failed
JPEGEncFeedFrame 1808: Stream flush failed with error = 8