Open gefend opened 2 years ago
To help you debug this, can you please provide the following information?
Got it, thanks for the info. May I ask how you are computing the results? Using different random seeds I was still able to get an accuracy of 60+
labels = df[gloria.constants.CHEXPERT_COMPETITION_TASKS].to_numpy().argmax(axis=1)
pred = similarities[gloria.constants.CHEXPERT_COMPETITION_TASKS].to_numpy().argmax(axis=1)
acc = len(labels[labels == pred]) / len(labels)
print(acc) # 0.607
The following is the code I used, I tried to replace the results calculation to the one you suggested but I get the same results. The only thing that I think I'm doing different than you is that I take only 200 images every run because of my GPU memory capacity.
import torch
import gloria
import pandas as pd
from gloria.datasets.preprocess_datasets import preprocess_chexpert_5x200_data
df = preprocess_chexpert_5x200_data()
df = df[0:200]
# load model
device = "cuda" if torch.cuda.is_available() else "cpu"
gloria_model = gloria.load_gloria(device=device)
cls_prompts = gloria.generate_chexpert_class_prompts()
# process input images and class prompts
processed_txt = gloria_model.process_class_prompts(cls_prompts, device)
processed_imgs = gloria_model.process_img(df['Path'].tolist(), device)
# zero-shot classification on 1000 images
similarities = gloria.zero_shot_classification(
gloria_model, processed_imgs, processed_txt)
labels = df[gloria.constants.CHEXPERT_COMPETITION_TASKS].to_numpy().argmax(axis=1)
pred = similarities[gloria.constants.CHEXPERT_COMPETITION_TASKS].to_numpy().argmax(axis=1)
acc = len(labels[labels == pred]) / len(labels) #0.17
Hi, I have the same issue with you. Have you figured it out?
I tried using your script for zero shot classification together with the pretrained weights (both resnet18 and resnet50). The calssification results I got are very random (accuracy 17-22% for each class). Maybe there is an aditional step needed or the weights I downloaded are not the trained weights?