Transfer Learning?

[trawler0]

If I my perception is correct, this repo is mostly about training models from scratch. However, for many applications, it might be best to fine tune/domain adapt a pretrained model on custom data. For example, one might stack a new head on top of a pretrained model, use a small learning rate with warmup for the original model and train the new model with sim-clr (or something else).

[guarin]

Hi! You are correct that the focus of the library is on training models from scratch. We haven't incorporated finetuning because there are many possible finetuning tasks (classification, detection, segmentation, tracking, etc.) and there are already good libraries for them. Instead, we try to keep lightly as simple as possible and make it easy to pretrain models for any downstream task. In most cases you should be able to save the state dict from the pretrained backbone and load the weights in your favorite finetuning library.

We also have some tutorials that include finetuning:

• classification: https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_moco_memory_bank.html
• object detection: https://docs.lightly.ai/self-supervised-learning/tutorials/package/tutorial_pretrain_detectron2.html

[trawler0]

Hey guarin! Thank you for the quick responses. Maybe I wasn't 100% precise with my comment. I wasn't talking about supervised fine-tuning, but about a self-supervised step in between. Something like that: https://arxiv.org/pdf/2103.12718 The use case is that many practitioners might not have enough data to train SSL from scratch, but still not enough annotators to label their entire data. So they might benefit from something like: download a pretrained model -> fine-tune self supervised on all their data -> fine-tune supervised on their labeled data. But I understand if this is not your focus right now, it's just something that is missing in many libraries in my opinion.

[guarin]

Ah sorry I didn't understand the question correctly! In principle everything described in the paper should already be possible with Lightly AFAIK (awesome paper btw!). If you have a pretrained model you can just continue pretraining on a different dataset. Adding heads should also be pretty straight forward (see classification tutorial from above). Are there any particular features that you would be interested in or are you mostly interested in some documentation on how to do transfer learning?

We also discussed hosting pretrained model weights in the past but didn't make them very easily available yet (weights are available here). Would this be something that you are looking for?

[trawler0]

No, I more or less wanted to engage with the community and share that I think this is an interesting rather overlooked area. Note however that it isn't 100% clear how to do this. For example, one could fine-tune the original model using LoRA or elastic weight consolidation (penalize the model for deviating from the original pretrained weigths).

[guarin]

This might be an interesting topic for a tutorial 🤔