Experiments to run

[jack89roberts]

Datasets

• Oxford Pets
• WikiArt
• RVL-CDIP
• Binary datasets (3?)

No. Samples (training and metrics)

• 500
• 1000
• 5000
• 10000
• 50000
• Where the full dataset is smaller than one of these, do a job with the whole dataset instead (unless that is very close to one of the other values)

Models

New models in bold

• apple/mobilevit-xx-small
• apple/mobilevit-small
• facebook/deit-tiny-patch16-224
• facebook/deit-small-patch16-224
• facebook/deit-base-patch16-224
• facebook/deit-base-patch16-384
• facebook/dinov2-small-imagenet1k-1-layer
• facebook/dinov2-base-imagenet1k-1-layer
• google/vit-base-patch16-384
• google/vit-large-patch16-224
• microsoft/beit-base-patch16-224
• microsoft/beit-base-patch16-384
• microsoft/beit-large-patch16-512
• microsoft/cvt-13
• microsoft/cvt-21-384
• microsoft/swin-base-patch4-window12-384
• microsoft/swin-large-patch4-window12-384-in22k
• microsoft/swinv2-base-patch4-window8-256
• microsoft/swinv2-large-patch4-window12to16-192to256-22kto1k-ft
• Zetatech/pvt-large-224

Seeds

As the above creates up to 100 training jobs per dataset I suggest we run with only one seed.

Metrics

• Renggli
• LogME
• No. parameters
• (ImageNet-1k validation accuracy - not part of jobs)
• ~(PARC? - likely to cause issues with larger samples due to size of correlation matrices)~