This could lead to training the linear probe with more data per class when considering the few-shot setting, because the code only filters the training data [Line 235] but not the validation dataset.
So at least for the few-shot setting, it would be necessary to use only the train_loader to train the classification head of the linear probe.
I noticed, that the linear probing is trained with both the validation and training dataset when using weight-decay tuning.
train_loader = feature_train_val_loaderhttps://github.com/LAION-AI/CLIP_benchmark/blob/a23028288ed914efb31c5ff32a1e702123cbd0ff/clip_benchmark/metrics/linear_probe.py#L280This could lead to training the linear probe with more data per class when considering the few-shot setting, because the code only filters the training data [Line 235] but not the validation dataset.
So at least for the few-shot setting, it would be necessary to use only the train_loader to train the classification head of the linear probe.