YOLOv5m/l-seg, which layer(s) should I take for feature extraction (object tracking) ?

[danwanban]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hi,

I am using Yolov5m-seg/Yolov5l-seg (later on converted to tensorrt on Jetson using tensorrtx repo).

I would like to take the output of a layer(s) as feature map per object, to be used as the input for a tracking module.

Which layer(s) should I use ? (preferably smallest in size and latest as possible in the flow) ?

Thanks for the help ! your work is awesome, really appreciate this repo.

Additional

No response

[github-actions[bot]]

👋 Hello @danwanban, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email support@ultralytics.com.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[AyushExel]

@mikel-brostrom is the expert here if you're looking for advice on tracking research. But if you just want to use an existing tracker, you can use this repo which is actively maintained by Mike https://github.com/mikel-brostrom/Yolov5_StrongSORT_OSNet

[mikel-brostrom]

Glad to help here @AyushExel. I would not recommend to use Yolov5 features for tracking. This is because the way in which Yolov5 is trained is not meant to learn detailed discriminative features.

An example: the shoe region of a detected person could give away that the person to its right isn't the same person in the case that one of the would be wearing sandals and the other one sneakers. The current loss functions is not meant to separate these instances in embedding space as much as possible. This is the basis for successful object re-identification, specially if you rely mostly on appearance descriptors and less on motion information.

If you still want to use Yolov5 features I would extract features from the deepest C3 module (20 x 20 x 1024) as this combines scaled high level features with low level features. You could also train a Yolov5 architecture with triplet loss instead of the current compound loss for the specific task of ReID.

[rukshankr]

@mikel-brostrom @AyushExel where can I find resources on how to extract features from YOLOv5? Newbie here, and I couldn't find a guide for that anywhere. Appreciate it if you could help

[github-actions[bot]]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

[danwanban]

(Apologies for the long delay) @mikel-brostrom - Thanks for the reply. A followup q:

• My goal is to use the features to do video object tracking. Now, at the nms stage, anything the best result is filtered. If I understand correctly (not sure about this), There is no guarantee that this result will be from the deepest C3 module - It can be from the other 2. So - If I take the features from the chosen C3 layer ( 256,512 or 1024) and would like to compare it to other frame features, I need to use the same feature vector size. To sum up, i need either:
• (A) Mark in the nms stage the feature vector from the deepest (1024) layer that correlates to the chosen feature vector by the nms, even if this is not the one that is the result of the nms stage (best prob)
• (B) Somehow equalize the dimensions of the selected feature vectors, if they are from different C3 layers. Should I use pooling in such a case ?
• (C) crop each item and use some other features extraction method - in this case, which generic one is recommended (quick + convertible to tensorrt) that does not require training ? (mobilenetv1/3 maybe? )

What are your thoughts on this ?

Thanks for all the help !