search

mlfoundations / open_clip

An open source implementation of CLIP.
Other
9.14k stars 908 forks source link

unlocking layer groups in open_clip/src/open_clip /timm_model.py for fine-tuning #854

Closed mikelee-dev closed 2 months ago

mikelee-dev commented 3 months ago

inside of open_clip/src/open_clip /timm_model.py, there is a function TimmModel.lock() which is stated to allow you to fine tune the model by unfreezing the weights in certain layer groups starting from the head of the model.

I was wondering what the layer groups explicitly refer to? so for example TimmModel.lock(unlocked_groups=1) vs. TimmModel.lock(unlocked_groups=2), and how to unlock all groups?

Thanks!

rwightman commented 2 months ago

@mikelee-dev groups are roughly blocks of associates params ... so stem (patch embed + pos embed) are group, each block is a group, the final norm is usually part of the last block (as a group) and head is last group.