search

insight-platform / Savant

Python Computer Vision & Video Analytics Framework With Batteries Included
https://savant-ai.io
Apache License 2.0
552 stars 45 forks source link

Module processes only one source when multi-stream adapter is restarted #495

Closed denisvmedyantsev closed 11 months ago

denisvmedyantsev commented 11 months ago

Module config

name: test
parameters:
  batch_size: 1

The first run of multi-stream adapter with 8 sources:

./scripts/run_source.py multi-stream --number-of-streams=8 data/jumanji_cast.mp4

Adapter sends 3725 frames x8, Module processes 8 sources, 29800 frames.

Module log fragment

 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-0
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-0 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-0: src_source-0.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-1
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-1 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-1: src_source-1.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-2
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-2 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-2: src_source-2.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-3
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-3 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-3: src_source-3.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-4
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-4 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-4: src_source-4.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-5
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-5 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-5: src_source-5.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-6
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-6 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-6: src_source-6.
 INFO  insight::savant::savant_rs_video_decode_bin  > Adding branch with source source-7
 INFO  insight::savant::savant_rs_video_decode_bin  > Branch with source source-7 added
 INFO  insight::savant::savant_rs_video_demux       > Created new src pad for source source-7: src_source-7.
 INFO  insight::savant::test                        > Added source source-1
 INFO  insight::savant::test                        > Added source source-0
 INFO  insight::savant::test                        > Added source source-3
 INFO  insight::savant::test                        > Added source source-2
 INFO  insight::savant::test                        > Added source source-7
 INFO  insight::savant::test                        > Added source source-6
 INFO  insight::savant::test                        > Added source source-4
 INFO  insight::savant::test                        > Added source source-5
 INFO  insight::savant::deepstream::buffer_processor > Processed 10000 frames, 670.40 FPS.
 INFO  insight::savant::deepstream::buffer_processor > Processed 10000 frames, 1461.77 FPS.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-0.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-1.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-2.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-3.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-4.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-5.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-6.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-7.
nvstreammux: Successfully handled EOS for source_id=6
nvstreammux: Successfully handled EOS for source_id=7
nvstreammux: Successfully handled EOS for source_id=5
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-4
nvstreammux: Successfully handled EOS for source_id=0
nvstreammux: Successfully handled EOS for source_id=3
nvstreammux: Successfully handled EOS for source_id=1
nvstreammux: Successfully handled EOS for source_id=2
nvstreammux: Successfully handled EOS for source_id=4
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-4 removed
 INFO  insight::savant::test                         > Resources for source source-4 has been released.
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-5
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-5 removed
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-6
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-6 removed
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-1
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-1 removed
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-2
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-2 removed
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-0
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-0 removed
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-3
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-3 removed
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-7
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-7 removed
 INFO  insight::savant::test                         > Resources for source source-7 has been released.
 INFO  insight::savant::test                         > Resources for source source-2 has been released.
 INFO  insight::savant::test                         > Resources for source source-3 has been released.
 INFO  insight::savant::test                         > Resources for source source-0 has been released.
 INFO  insight::savant::test                         > Resources for source source-1 has been released.
 INFO  insight::savant::test                         > Resources for source source-6 has been released.
 INFO  insight::savant::test                         > Resources for source source-5 has been released.

The second run (each subsequent run):

./scripts/run_source.py multi-stream --number-of-streams=8 data/jumanji_cast.mp4

Adapter sends 3725 frames x8, Module processes 1 source, 3725 frames.

 INFO  insight::savant::savant_rs_video_decode_bin   > Adding branch with source source-0
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-0 added
 INFO  insight::savant::savant_rs_video_demux        > Created new src pad for source source-0: src_source-0.
 INFO  insight::savant::test                         > Added source source-0
 INFO  insight::savant::deepstream::buffer_processor > Processed 10000 frames, 709.11 FPS.
 INFO  insight::savant::savant_rs_video_demux        > Received EOS from source source-0.
nvstreammux: Successfully handled EOS for source_id=8
 INFO  insight::savant::savant_rs_video_decode_bin   > Removing branch with source source-0
 INFO  insight::savant::savant_rs_video_decode_bin   > Branch with source source-0 removed
 INFO  insight::savant::test                         > Resources for source source-0 has been released.
tomskikh commented 11 months ago

Zeromq_src skips the frames from the other sources.

 DEBUG insight::savant::utils::zeromq                                   > Routing ID for topic b'source-0/' changed from b'\x00k\x8bEg' to b'\x00k\x8bEh'.
 DEBUG insight::savant::zeromq_src::zeromq_src+zeromqsrc0               > Received message of sizes [488, 127383]
 DEBUG insight::savant::zeromq_src::zeromq_src+zeromqsrc0               > Receiving next message
 DEBUG insight::savant::utils::zeromq                                   > Skipping message from topic b'source-1/': routing ID b'\x00k\x8bEh', expected b'\x00k\x8bEg'.
 DEBUG insight::savant::zeromq_src::zeromq_src+zeromqsrc0               > Received message of sizes [488, 7879]
denisvmedyantsev commented 11 months ago

resolved with #506