search

hushell / pmf_cvpr22

181 stars 23 forks source link

How to get unknown class? #29

Closed gigasurgeon closed 3 months ago

gigasurgeon commented 1 year ago

I have 5 classes, but during inference time there is no guarantee that the image will be from these 5 classes. I am using dino_small_patch16 arch. Is there a way to identify an outlier image, or an unknown class?

westbalon commented 11 months ago

@gigasurgeon hello. did you manage to identify outlier image and make a predictor using this repo? I would appreciate any help

gigasurgeon commented 11 months ago

@gigasurgeon hello. did you manage to identify outlier image and make a predictor using this repo? I would appreciate any help

@westbalon I generated embeddings for 5 images per class and saved it as .pt files locally. Then, for each test image, I generated embedding and compared it with the vectors already stored, using cosine similarity. It gave around 85% accuracy in my usecase.

I got greater success with CLIP model using this approach. I used its image embeddding as well as the text embeddings and added them together and saved them locally. The text embedding was generated with either some words written on the object or a sentence describing the object. During test time, I generated only image embedding through CLIP and compared the cosine similarity. If it was below a certain threshold, I marked them as unknown or outlier. This gave me 100% accuracy for my usecase. You can use huggingface's openAI clip (giant variant worked best).