Issue with inference values with 'hf-hub:laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg'

[dbugger29]

Can you help me with the following:

I receive different values for the same image and with the same class names.

import torch
import pickle
from PIL import Image
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms('hf-hub:laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg')
tokenizer = open_clip.get_tokenizer('hf-hub:laion/CLIP-convnext_xxlarge-laion2B-s34B-b82K-augreg')
image = preprocess(Image.open("<some_image>")).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat"])
with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)
    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)  # prints: [[1., 0., 0.]]

These are the results after I ran the same code twice:

Label probs: tensor([[4.4378e-04, 6.0942e-01, 3.9013e-01]])
Label probs: tensor([[3.7052e-04, 6.6962e-01, 3.3001e-01]])

This does not seems to be the case with other models. Any help would be appreciated 🙇‍♀️

[rwightman]

@dbugger29 you are right, this had me puzzled for a second, you need model.eval() ... open_clip models default to .train() mode like many torch model libs. Why is this needed? convnext doesn't have batchnorm like the Resnet models, but it does have stochastic depth active by default which will inject randomness in train mode.

[dbugger29]

Thank you :D

[dbugger29]

Hello, @rwightman, I come back with a similar issue with: ViT-H-14-378-quickgelu. Can you help me with this also? Also, I am trying the latest models from https://github.com/facebookresearch/MetaCLIP/, I cannot load: open_clip.create_model_and_transforms('ViT-bigG-14-quickgelu', pretrained='metaclip_fullcc') Any kind of help would be appreciated :D thank you