Test set for Zero-shot Benchmark Datasets

[rajanieprabha]

Hi,

For zero-shot classification results, some datasets don't have specific test sets. In order for us to compare RemoteCLIP and reproduce the results, do you plan to provide more information on the test sets of various datasets? It would definitely be helpful. Thanks.

[youngbaldy]

Since some remote sensing classification datasets do not have official test sets, we adopt an 8:2 train/test split ratio for our zero-/few-/full-shot classification experiments on RemoteCLIP. This follows the common practice in related studies, such as “Do We Still Need ImageNet Pre-Training in Remote Sensing Scene Classification?” and “An Empirical Study of Remote Sensing Pretraining,” among others. Specifically, for datasets without an official test split, we sort the images within each class by file name. We then use the first 80% of the images as the training set and the remaining 20% as the test set. To further facilitate comparison of experimental results, we are planning to prepare and release the test set file name list in the near future.

[rajanieprabha]

That's helpful. Thank you!