search

keyu-tian / SparK

[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
https://arxiv.org/abs/2301.03580
MIT License
1.41k stars 82 forks source link

Can SparK be used for few-shot learning? #77

Closed Y-T-G closed 6 months ago

Y-T-G commented 6 months ago

Like SwaV, is it possible to fine-tune SparK pretrained models on a very small subset of the dataset and get good accuracy, or is it not intended for that?

keyu-tian commented 6 months ago

@Y-T-G Although I didn't try such a low-shot finetuning, I believe it is hopeful to get a decent accuracy, because when finetuing ConvNeXt-Small, I observed the accuracies of the first three finetuning epochs were 28.4, 53.7, and 61.9.

Y-T-G commented 6 months ago

@keyu-tian I see. Alright.