Question about fine-grained dataset CUB and Aircraft

sgvaze / osr_closed_set_all_you_need

Official repo for our ICLR 22 Oral paper: "Open-Set Recognition: a Good Closed-Set Classifier is All You Need?"

MIT License

261 stars 32 forks source link

Question about fine-grained dataset CUB and Aircraft #18

Closed Hrren closed 2 years ago

Hrren commented 2 years ago

Hello When I tried to reproduce the AUC results of the Easy and Medium datasets for the CUB and Aircraft datasets, I couldn't reproduce the results of the paper, for CUB, Easy and Medium were only 87.5 and 81.8; for Aircraft is 89.0 and 85.4. I trained according to the bash script/osr_finegrained_train.sh and fix corresponding Optimal Hyper-parameters, and used the places_moco pre-trainmodel, but the results in the paper could not be reproduced. What is the reason?

sgvaze commented 2 years ago

Hi, I imagine the reason for the discrepancy with the 'Medium' examples is that in the latest version, we combine 'Medium' and 'Hard' into a single split. Regarding 'Easy', I am not sure exactly the reason for the discrepancy (we fix the seeds at the start of training), but within 1% AUROC may be expected between different runs.