search

huggingface / pytorch-image-models

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more
https://huggingface.co/docs/timm
Apache License 2.0
31.79k stars 4.72k forks source link

[BUG] where are the OOD results? #2259

Closed tfriedel closed 1 month ago

tfriedel commented 1 month ago

Describe the bug I noticed this recent commit: https://github.com/huggingface/pytorch-image-models/commit/4515a435e4844d01269eac111ff67e5bcf5bbc6e

It says in the results folder there are some results for OOD images. I couldn't find them. I was a bit unsure if maybe the sketches or adversarial images could have been meant as OOD? Or if there's actually some results for classes that are not part of the 1000 imagenet classes, but maybe they were not commited.

rwightman commented 1 month ago

@tfriedel sketch, rendition (r), I consider to be quite out-of-distribution. The classes need to line up as these aren't open-set models / zero-shot models, but if you look at them they are quite different than imagenet images. ImageNet-v2 is also since it was a separate sampling of images, but there the goal was to try and match the selection criteria of imagenet in various ways though with a different pool of (more recent) images.

rwightman commented 1 month ago

If you want to classify or cluster/search images outside the imagenet classes, I've long wanted to do a retrieval benchmark but it's quite involved so hasn't topped my priority list. But contributions welcome.