How to deactivate Deep contribution to DeepSORT

[jakobGTO]

Hello,

Is there a simple way to run the tracking without a re-id network that extracts features? if one would like to determine how much of an increase the re-id networks make in terms of performance compared to not extracting any features at all

Thanks

[jakobGTO]

My current idea is to simply replace

self.feature = np.asarray(feature.cpu(), dtype=np.float32)

in detection.py with

self.feature = np.zeros_like(feature.cpu(), dtype=np.float32)

would this be a correct approach?

[mikel-brostrom]

would this be a correct approach?

The correct approach would be to set _lambda=1 here:

https://github.com/mikel-brostrom/Yolov5_DeepSort_OSNet/blob/8885642c9d049c933c6e1df1d05478dab4a0c37c/deep_sort/sort/tracker.py#L38

[youonlytrackonce]

You may have a look at SORT algorithm.

[mikel-brostrom]

Setting _lambda=1 would be like using only SORT @youonlytrackonce as it zeroes out the deep description appearance and only takes into consideration the KF predictions

[jakobGTO]

Thanks a lot!

Does this mean the re-id network is still used, just the deep contribution vector is weighted to 0? What I'm asking is basically will the inference time be reduced as expected in the SORT algorithm when not using deep contribution?

[mikel-brostrom]

will the inference time be reduced as expected in the SORT algorithm when not using deep contribution?

No, the Deep descriptor will still be computed. But ignored

[yamiefun]

Hello, i'm still curious about how to skip re-id network? I just want to know the tracking performance without getting features. I tried to modify the _get_feature function in strong_sort.py like features = torch.zeros([len(im_crops), 512], dtype=torch.float32) instead of features = self.model(im_crops) but get error as below:

File "strong_sort/sort/linear_assignment.py", line 55, in min_cost_matching
    row_indices, col_indices = linear_sum_assignment(cost_matrix)
ValueError: matrix contains invalid numeric entries

Thanks.

[mikel-brostrom]

The tracking method got updated. It is not DeepSORT anymore. So the new way of doing this changed as well. This:

https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/blob/07ae91cc30f03072cf702206e88053185d144eaa/strong_sort/sort/linear_assignment.py#L173

Should be:

cost_matrix[row] = 0 * cost_matrix[row] + 1 *  gating_distance 

for no appearance description. Let me know what results you get if you plan to evaluate it this way :smile:

[yamiefun]

Hi @mikel-brostrom , Thanks for fast reply! I modified the code like you did, but still get the same error if I tried to use features = torch.zeros([len(im_crops), 512], dtype=torch.float32) to skip re-id network inferencing.

  File "strong_sort/sort/tracker.py", line 85, in update
    self._match(detections)
  File "strong_sort/sort/tracker.py", line 159, in _match
    linear_assignment.matching_cascade(
  File "strong_sort/sort/linear_assignment.py", line 123, in matching_cascade
    min_cost_matching(
  File "strong_sort/sort/linear_assignment.py", line 55, in min_cost_matching
    row_indices, col_indices = linear_sum_assignment(cost_matrix)
ValueError: matrix contains invalid numeric entries

[mikel-brostrom]

Sorry, cannot reproduce. This works for me

 cost_matrix[row] = 0 * cost_matrix[row] + (1) *  gating_distance

[mikel-brostrom]

These are the results:

HOTA: exp329-pedestrian            HOTA      DetA      AssA      DetRe     DetPr     AssRe     AssPr     LocA      RHOTA     HOTA(0)   LocA(0)   HOTALocA(0)
MOT16-02                           28.761    34.078    24.59     36.884    71.332    27.889    50.437    80.91     30.015    36.404    74.675    27.184    
MOT16-04                           53.513    57.12     50.751    62.426    75.467    54.127    76.761    81.17     56.186    70.226    75.402    52.952    
MOT16-05                           33.121    35.945    30.649    38.837    72.985    38.135    61.584    81.818    34.48     42.183    76.453    32.25     
MOT16-09                           43.21     55.803    33.526    65.882    72.245    40.438    58.842    85.368    46.994    53.225    81.066    43.148    
MOT16-10                           31.941    44.586    23.164    50.352    69.015    27.71     55.268    80.31     34.074    42.454    74.079    31.449    
MOT16-11                           56.614    58.964    54.647    70.934    72.823    62.822    76.882    86.852    62.226    67.116    83.392    55.97     
MOT16-13                           36.746    38.205    35.732    42.812    67.857    41.96     57.28     80.147    39.025    48.487    73.636    35.704    
COMBINED                           45.365    48.845    42.764    54.334    73.042    47.554    69.038    81.842    48.045    58.226    76.172    44.352    

CLEAR: exp329-pedestrian           MOTA      MOTP      MODA      CLR_Re    CLR_Pr    MTR       PTR       MLR       sMOTA     CLR_TP    CLR_FN    CLR_FP    IDSW      MT        PT        ML        Frag      
MOT16-02                           35.782    78.145    39.584    45.646    88.277    14.815    55.556    29.63     25.806    8140      9693      1081      678       8         30        16        653       
MOT16-04                           66.442    78.505    67.656    75.188    90.895    45.783    38.554    15.663    50.281    35757     11800     3582      577       38        32        13        790       
MOT16-05                           39.205    79.319    41.772    47.492    89.25     16        60        24        29.383    3238      3580      390       175       20        75        30        208       
MOT16-09                           56.154    83.947    59.54     75.366    82.645    48        52        0         44.055    3962      1295      832       178       12        13        0         173       
MOT16-10                           48.531    77.217    53.475    63.216    86.647    27.778    62.963    9.2593    34.128    7787      4531      1200      609       15        34        5         723       
MOT16-11                           60.889    85.479    62.372    79.889    82.017    50.725    39.13     10.145    49.289    7329      1845      1607      136       35        27        7         181       
MOT16-13                           40.769    76.874    45.135    54.114    85.77     24.299    43.925    31.776    28.254    6196      5254      1028      500       26        47        34        434       
COMBINED                           54.196    79.226    56.78     65.584    88.165    29.787    49.903    20.309    40.572    72409     37998     9720      2853      154       258       105       3162      

Identity: exp329-pedestrian        IDF1      IDR       IDP       IDTP      IDFN      IDFP      
MOT16-02                           35.322    26.793    51.817    4778      13055     4443      
MOT16-04                           64.861    59.257    71.636    28181     19376     11158     
MOT16-05                           43.462    33.294    62.569    2270      4548      1358      
MOT16-09                           49.786    47.594    52.19     2502      2755      2292      
MOT16-10                           37.418    32.359    44.353    3986      8332      5001      
MOT16-11                           65.566    64.716    66.439    5937      3237      2999      
MOT16-13                           46.91     38.253    60.631    4380      7070      2844      
COMBINED                           54.051    47.129    63.356    52034     58373     30095     

Count: exp329-pedestrian           Dets      GT_Dets   IDs       GT_IDs    
MOT16-02                           9221      17833     111       54        
MOT16-04                           39339     47557     178       83        
MOT16-05                           3628      6818      163       125       
MOT16-09                           4794      5257      61        25        
MOT16-10                           8987      12318     167       54        
MOT16-11                           8936      9174      159       69        
MOT16-13                           7224      11450     121       107       
COMBINED                           82129     110407    960       517

[mikel-brostrom]

	HOTA(↑)	MOTA(↑)	IDF1(↑)
With visual features	54.087	61.268	68.563
Without visual features	45.365	54.196	54.051

[yamiefun]

Have you tested the speed W/ and W/O visual features?

[mikel-brostrom]

By doing this: https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/issues/386#issuecomment-1265280715 the appearance descriptor is still calculated, just ignored in the association process. So, no I have not tested the speed. But feel free to drop your experiment results here :smile:

[yamiefun]

I just can't find a proper way to skip reid model. Like I posted above, I tried to use torch.zeros as fake features but failed. Do you have any suggestion? Thanks.

[mikel-brostrom]

This works for me:

self.feature = np.ones([512], dtype=np.float32)

But it is still not the right way to deactivate the visual appearance descriptor. This is: https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet/issues/386#issuecomment-1265280715