Some questions about the CUB dataset

[judgment404]

Thanks for your work! I attempted to train on CUB and installed kmeans_pytorch directly from the GitHub repository https://github.com/subhadarship/kmeans_pytorch. The final results were:

Epoch: 198 Avg Loss: 4.103 | Contrastive: 4.103 
Testing on disjoint test set...
Running k-means on cuda:0..
Model saved to osr_novel_categories/metric_learn_gcd/log/(16.10.2024_|_06.586)/checkpoints/model.pt.
Best Train Epochs:  160
Epoch None, SS-K-Means Train ACC Unlabelled_v1: All 0.7117 | Old 0.7505 | New 0.6924
Epoch None, SS-K-Means Train ACC Unlabelled_v2: All 0.6951 | Old 0.7185 | New 0.6834
Hierarchy clustering
Epoch: 199 Avg Loss: 4.087 | Contrastive: 4.087 
Testing on unlabelled examples in the training data...
Running k-means on cuda:0..
Testing on disjoint test set...
Running k-means on cuda:0..
Performing SS-K-Means on all in the training data...
Performing SS-K-Means on all in the training data...
Train Accuracies: All 0.6450 | Old 0.6338 | New 0.6507
Test Accuracies: All 0.6048 | Old 0.6103 | New 0.5993
Model saved to osr_novel_categories/metric_learn_gcd/log/(16.10.2024_|_06.586)/checkpoints/model.pt.
Best Train Epochs:  160
############# Final Reports #############
Best Train Accuracies: All 0.7166 | Old 0.7472 | New 0.7014
Best Train Epochs:  160 
Reports for the best checkpoint:
Reports for the last checkpoint:
osr_novel_categories/metric_learn_gcd/log/(16.10.2024_|_06.586)/checkpoints/model.pt

I tried to change the seed and ran it 5 times, but I got similar numbers: 71 or 70, which are lower than the number in the paper (73). I also tried to train on the aircraft dataset, but the number I got was 51, which is also lower than the paper's number (57). I trained using the default hyperparameters in contrastive_train.sh, and I’m not sure if I misconfigured the training parameters or something else.

Do you have any suggestions on how I might address this issue? Additionally, would it be possible to access your training weights? Thank you!

[SarahRastegar]

Thank you for your interest in our work! Here are a few notes to help you replicate our results:

Use the Correct Training Script: Run the training using the following command to ensure you're using the same hyperparameters as in our paper:

1. Use the Python Training Script: Run the training using the following command in folder SelEx/methods/contrastive_training to ensure you're using the same hyperparameters as in our paper:

   python contrastive_training.py

2. PyTorch Version Matters: Please make sure you're using PyTorch version 2.x, as specified in the requirements.txt. Using lower versions like 1.x leads to reduced performance.
3. GPU Architecture: Our results were obtained using NVIDIA A6000 GPUs. While I've achieved consistent results on different machines with A6000 GPUs, I noticed some variations when using different GPU architectures.
4. Dataset Specifics: If you're working with the CUB dataset, set the unsupervised_smoothing parameter to 1.0, for other fine-grained datasets (Scars, Pets, Aircraft) to 0.5 and for generic datasets to 0.1.

   --unsupervised_smoothing 1.0

   Feel free to reach out if you have any more questions or need further assistance!

[SarahRastegar]

Last and Best Epoch Evaluations: I also noticed that the reports for the best and last epochs in the final output are empty. This suggests that in the following lines of code

print("Reports for the best checkpoint:")
        os.system("CUDA_VISIBLE_DEVICES="+str(args.gpu_id)+" python ../clustering/extract_features.py --dataset "+args.dataset_name+
                  " --warmup_model_dir "+ args.model_path.replace('(','\(').replace(')','\)').replace('|','\|'))
        os.system("CUDA_VISIBLE_DEVICES="+str(args.gpu_id)+" python ../clustering/k_means.py --dataset "+args.dataset_name+
                  " --unbalanced "+str(int(args.unbalanced)))

It seems that the scripts extract_features.py and the subsequent k_means.py are not executing properly. This might be due to insufficient permissions to run OS commands from within the Python script. As a workaround, please run these two steps manually using the checkpoint address printed at the end of the reports.

Let me know if this solves the problem.