Self-supervised initialization

[maduc7]

Hi,

I really liked your paper. Great job! I was trying to replicate it for my work, and I am not sure how you are initializing the SimCLR network for self-supervision training. Did you train it from scratch?

Thanks a lot.

[ozanciga]

from scratch so the way it's done here: https://github.com/pytorch/vision/blob/main/torchvision/models/resnet.py

if you initialize a resnet in pytorch (without specifying pretrained=True, ie pretrained=False) you will start from our starting point.

also, see this work which can help get you a better performance overall: https://arxiv.org/abs/2109.01721

without going into it too much, you can just start from pretrained=True and do everything else the same, and you will get a better final performance if you train long enough (if you want to train for shorter you can apply the techniques described in above referenced paper)

[maduc7]

Thanks a lot for your answer. One last question, does that mean that for the weights you are sharing there https://github.com/ozanciga/self-supervised-histopathology/releases/tag/tenpercent you used pretrained=True in order to reach a better overall performance?

[ozanciga]

Thanks a lot for your answer. One last question, does that mean that for the weights you are sharing there https://github.com/ozanciga/self-supervised-histopathology/releases/tag/tenpercent you used pretrained=True in order to reach a better overall performance?

no this work (the weights you linked) predates the arxiv paper i posted above. so pretrained=False for https://arxiv.org/abs/2011.13971