How to use Deep SORT in my own project?

[LAWSSSS]

I've used the SORT following this instruction: https://github.com/abewley/sort#using-sort-in-your-own-project I wonder can I use a similar way in my own project like I did with the SORT?

[ozanpkr]

I've used the SORT following this instruction: https://github.com/abewley/sort#using-sort-in-your-own-project I wonder can I use a similar way in my own project like I did with the SORT?

Hi brother.Can you help me about on Sort algorithm.I use YOLOv2 and ı couldn't use sort like you.

[LAWSSSS]

@ghost-ronin Sorry for being late. I just follow the instruction provided in the SORT GitHub which is the following:

from sort import *
#create instance of SORT
mot_tracker = Sort() 

# get detections
...

# update SORT
track_bbs_ids = mot_tracker.update(detections)

# track_bbs_ids is a np array where each row contains a valid bounding box and track_id (last column)

[WiiliamC]

I have the same issue. Hope to use Deep Sort in my own project.

[diego0718]

Hi all,

Good morning i was wonder if someone has tried deep sort with proper dataset ( In my case i trained human heads). When i am doing inference over a video i get the bboxes and the confindence. How can i pass these variables to deep sort? i have to train something more to initialize Deep sort? i am very confused with this. The id swiching problem is adressed? ( i had probed SORT )

Thank u

[Christoph9402]

Hi, I'm also having trouble implementing deepsort in my own project. Just like you, I did manage to run SORT, but I'm struggeling with DeepSort. Did you find any solution?

[ozanpkr]

Hi, I'm also having trouble implementing deepsort in my own project. Just like you, I did manage to run SORT, but I'm struggeling with DeepSort. Did you find any solution?

I did not find a solution. So I didn't DeepSort in my project.

[danielcrane]

@Christoph9402 @ozanpkr What were you guys struggling with in particular?

[ilinii]

any updates ?

[danielcrane]

any updates ?

Is there something in particular you're struggling with?

[prasuchit]

Hi @danielcrane, I was just wondering if there's a readme we can follow for training it on a custom dataset. Since the pretrained model can't detect all the custom objects I need, I'll need to finetune it for my dataset. Thanks.

[danielcrane]

@prasuchit Which model in particular? The detection model or the feature embedding model?

I personally use a version of YOLO (i.e. YOLOv7) for detection, and train a feature embedding model using pytorch-metric-learning.

You don't have to restrict yourself to the cosine metric learning model that's mentioned in the readme at all, if there's something else out there that better fits your needs.

[prasuchit]

Hi @danielcrane , thanks for getting back to me. I tried using the Yolo v7 + deep sort combo from this package, and saw that deep-sort lost track of the object if it moved fast and assigned a new label to it.

I have a pick and place task that I'm trying to accomplish with a cobot and I need a camera to detect and track similar objects on a conveyor belt. When the robot picks these objects up from the moving conveyor, they're going to move fast and I was wondering if I could train deep-sort to better track them. I've been using yolov5 for a couple of years now but I'm very new to deep-sort. Kindly let me know. Thanks.

[danielcrane]

@prasuchit I'd first begin by running YOLOv7 alone without Deep SORT to see if the problem actually lies with YOLOv7 being unable to detect the objects, or if it's an issue with Deep SORT itself.

If the issue is with Deep SORT and not the detection model, then it's mostly likely an issue with either:

• Feature embedding model
• Kalman filter

Debugging these is a bit more tricky, but I'd start with the feature embedding model by comparing the cosine distance between the feature embeddings of the same object at different times - if the same object is giving wildly different feature embeddings depending on where it lies on the conveyor (for example), then you would be best getting a custom dataset and training your own feature embedding model using i.e. pytorch-metric-learning

[vocdex]

@danielcrane Thanks for your suggestions. I am working on a similar problem. I trained Detectron2 instance segmentation model ( with bbox prediction) and used SORT to track objects ( single class, a plant in greenhouse). Detectron2 model works well but Kalman filter fails to track detected bboxes since the frames change quite fast sometimes ( every 4th-5th second) and the linear motion model doesn't apply there. I was thinking of using DeepSORT to compensate for that sudden change in frames. Do you think DeepSORT can handle such sudden changes because of its extra appearance-based tracking?

[danielcrane]

@vocdex If the change is extremely dramatic (i.e. moving a whole frame width from one frame to the next) then Deep SORT would still struggle by default since the implementation here does still first check that the candidate is within a "reasonable" distance of the target (see the use of gating_distance below).

https://github.com/nwojke/deep_sort/blob/280b8bdb255f223813ff4a8679f3e1321b08cdfc/deep_sort/linear_assignment.py#L144-L190

It's a bit hard to tell if this'd cause you issues in your case without just trying it out, but you could always modify this behaviour if you find a more suitable problem-specific solution!

Just as a note, you would have to train (or use a pre-trained) network that can generate meaningful feature vectors in order to move from SORT to DeepSORT. If your object type is contained within the imagenet dataset then you could probably use the penultimate layer of ResNet-50 or something and treat that as a feature vector just to get a feel for things.

[vocdex]

@danielcrane Thanks for such a detailed response. I think the changes are not that fast. It almost looks like a few frames are skipped. I have used pre-trained Resnet50 and CLIP based embedders but the performance was not good as I expected. Next, I'll have to train Siamese network or use pytorch-metric-learning for embedding network training as you mentioned above.Thanks!

[Sata-Chakra]

any solution to this?

I am desperately trying to implement deepsort but the results are very poor. ALso the track.isconfirmed() is always returning false for my use case, hence I am not able to track anything properly before say 2-3 frames. ANy idea how to solve this ?