search

wisdomikezogwo / quilt1m

[NeurIPS 2023 Oral] Quilt-1M: One Million Image-Text Pairs for Histopathology.
https://quilt1m.github.io/
MIT License
138 stars 8 forks source link

Which preprocess should I use for linear probing ? #24

Closed anhtienng closed 7 months ago

anhtienng commented 8 months ago

Hi, thank you for your work.

I am adapt your model to my dataset,

  1. using preprocess_train for linear probing (only use the vision encoder); preprocess_val for testing.

_, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:wisdomik/QuiltNet-B-32') Is it correct ?

  1. Should I skip a projection layer of the vision model (the one maps features from 768 to 512), replace it by 768 -> num_class ?
wisdomikezogwo commented 8 months ago

  1. preprocess_train is for (pre)-training yes, and preprocess_val is for validation and evaluation yes.

  2. I'd test out what works best for your dataset, I'd test out adding a head so 512 -> N_classes, or skipping that layer and getting features from deeper layers depending on the needs, keep the other parameters the same, and eval to see what works better.

Thanks.

anhtienng commented 7 months ago

Thanks