How would you recommend running inference on multiple streams?

[umairjavaid]

For now I am running 6 dockers in parallel - one for each stream, and it is working fine. But, when I try to run 7 streams in parallel, a couple of containers fail to do inference and throw the following error. So, my question is, how would you recommend doing inference on multiple streams using your pipeline? Is my approach optimal?

error:

[TensorRT] ERROR: ../rtSafe/cuda/cudaConvolutionRunner.cpp (457) - Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
/workspace/FastMOT/fastmot/feature_extractor.py:74: RuntimeWarning: invalid value encountered in true_divide
  embeddings /= np.linalg.norm(embeddings, axis=1, keepdims=True)
[TensorRT] ERROR: ../rtSafe/cuda/cudaConvolutionRunner.cpp (457) - Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
[TensorRT] ERROR: ../rtSafe/cuda/cudaConvolutionRunner.cpp (457) - Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
[TensorRT] ERROR: ../rtSafe/cuda/cudaConvolutionRunner.cpp (457) - Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
[TensorRT] ERROR: ../rtSafe/cuda/cudaConvolutionRunner.cpp (457) - Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
2021-08-31 04:13:14 [    INFO] Found:        head   1 at ( 359, 132)
[TensorRT] ERROR: ../rtSafe/cuda/cudaConvolutionRunner.cpp (457) - Cudnn Error in execute: 8 (CUDNN_STATUS_EXECUTION_FAILED)
[TensorRT] ERROR: FAILED_EXECUTION: std::exception
Traceback (most recent call last):
  File "app.py", line 118, in <module>
    main()
  File "app.py", line 89, in main
    mot.step(frame)
  File "/workspace/FastMOT/fastmot/mot.py", line 141, in step
    self.tracker.update(self.frame_count, detections, embeddings)
  File "/workspace/FastMOT/fastmot/tracker.py", line 272, in update
    track.add_detection(frame_id, next_tlbr, (mean, cov), embeddings[det_id], is_valid)
  File "/workspace/FastMOT/fastmot/track.py", line 190, in add_detection
    self.avg_feat.update(embedding)
  File "/workspace/FastMOT/fastmot/track.py", line 109, in update
    self._average(self.sum, self.avg, embedding, self.count)
ZeroDivisionError: division by zero

[shubh7g]

@umairjavaid you can create any number of streams (7 in your case) fastmot.VideoIO objects as suggested in https://github.com/GeekAlexis/FastMOT/issues/157#issuecomment-900110539.

[umairjavaid]

I want to do inference in parallel for better performance. Creating 7 streams would make the inference sequential, no?

[GeekAlexis]

129 you can do multiprocessing.