ultralytics / yolov5

YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
https://docs.ultralytics.com
GNU Affero General Public License v3.0
49.83k stars 16.13k forks source link

Features map #2613

Closed GiuliaCiaramella closed 3 years ago

GiuliaCiaramella commented 3 years ago

❔Question

is there a feature map in yolov5 whose output is independent on image size?

Additional context

Hello :) I am interested in extracting the feature map from last layer in the head of the network, in particular, one of the three that goes to feed the Detect() layer. I want this feature map to produce a feature vector representing the input image. The problem I am facing, is that this vector depends on the input image size, and since my goal si to compare images starting from their feature vectors, I can't accomplish my goal if the two images have different sizes.

For now, in order to make things work, I am using VGG16() neural network to process the images and take the feature vector in the last layer before classification, which has always a lenght of 4096, since it depends on the network layer and not on the input image size.

However, since I am already using yolov5 for detection, that would be much more efficient extract the feature vector from yolov5 instead of reprocessing the image in another NN.

So my question is: is there a feature map in yolov5 whose output is independent on image size? I hope that this is clear.

github-actions[bot] commented 3 years ago

👋 Hello @GiuliaCiaramella, thank you for your interest in 🚀 YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Status

CI CPU testing

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

zonguu commented 3 years ago

As I concered, the true length of vector is (number of class + 5). The size of input influences the the number of vector , for example, 20^2 + 40^2 + 80^2. You can compare them with the given position , layers and anchors.

GiuliaCiaramella commented 3 years ago

@zonguu hey, thanks for the answer. I think I don't complete understand it. What are those numbers that you reported? how should I take them?

What I am sure by now, is that the smallest feature map before the detect layer has size= [1,3,8,13,9] for an image size of 416*256. 1 and 3 are always there (I think they represent 1=batch, 3 like channels of colors, but not sure), and 9 =4classes in my model+5. But the lenght in third and fourth position (number 8 and 13) change with image size.

glenn-jocher commented 3 years ago

@GiuliaCiaramella you can always use a nn.AdaptiveAvgPool2d() layer to average the spatial dimensions away, no matter what size they are. Most classification models like VGG do this, which is why their output vector is unaffected by image size. We have a Classify() module that does this as well here that you can use an example of how to get started (i.e. and if you want you can also apply it to each of the 3 Detect() layer inputs to get equal-size feature vectors.)

https://github.com/ultralytics/yolov5/blob/005d7a8c54a39d89bf2b9dc03fba82a489cd0628/models/common.py#L305-L315

GiuliaCiaramella commented 3 years ago

@glenn-jocher Amazing! Thank you, I will focus on it :)

kimngoc99 commented 3 years ago

@GiuliaCiaramella Hello, you have solved the above problem. And you can help can I convert feature map to feature vector. I can't do that

glenn-jocher commented 3 years ago

@GiuliaCiaramella good news 😃! Feature map visualization was added ✅ in PR #3804 by @Zigars today. This allows for visualizing feature maps from any part of the model from any function (i.e. detect.py, train.py, test.py). Feature maps are saved as *.png files in runs/features/exp directory. To turn on feature visualization set feature_vis=True in the model forward method and define the layer you want to visualize (default is SPP layer).

https://github.com/ultralytics/yolov5/blob/20d45aa4f18d7df7148fc6dd69a9c0607003f004/models/yolo.py#L158-L160

To receive this update:

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

layer_8_SPP_features