knowledge-transfer/

[utterances-bot]

Knowledge Transfer in Self Supervised Learning

A general framework to transfer knowledge from deep self-supervised models to shallow task-specific models

http://amitness.com/knowledge-transfer/

[israrbacha]

Hi Amit, I read your brief and interesting post about semi-supervised learning, would you suggest me to continue with this area or GANs for PhD research?. My master's background is from GANs but I found this area very interesting. will appreciate your suggestion Thanks!

[amitness]

@israrbacha A Ph.D. research topic is something you would spend years on and is a very personal choice. So I don't think I can give you a definite answer.

That said, semi and self-supervised learning are indeed active research areas right now. Maybe you could explore existing literature that utilizes GANs for semi/self-supervised learning and see if that interests you. That way you can combine your existing background of GANs with your interest.

[timothyvanco]

Hi Amit, thanks for great article. I am working on my diploma thesis in the field of Self-supervised learning and I would like to ask, if you know about any code how to do transfer learning to downstream task. I read a lot about pretext-tasks (also on your other article) so now I know some ways how they work, but I would like to know, how to transfer that knowledge into downstream task (probably to instance segmentation), but can be also any other. I would appreciate code example to understand it better. Because I read about theory in this article :)

So to sum it up - do you know a way how to do pretext task and then downstream task with aim to do instance segmentation (for example to mask-r-cnn) in code? Thanks!

[amitness]

@timothyvanco Yeah, there is pytorch-lightning-bolts. They've implemented most standard models for self-supervised learning.

I haven't worked on instance segmentation problems, so not sure. But, I think you can use it by swapping the backbone network with an SSL model and see how much performance improvement that gives.

[ramdhan1989]

Hi Amit, thanks for great article. Currently, I am playing around with jigsaw puzzles in this paper https://arxiv.org/pdf/1603.09246v3.pdf. After finishing pretext task, how to do downstream task using labeled dataset for classification task ? do we need to make 9 patches of image or just pass the whole images ?

[amitness]

@ramdhan1989 For downstream task, the paper initialized AlexNet with weights from the pre-text task and used that for classification. We don't need to make 9 patches.

See the bolded part of comment of Figure 3 in the paper.

[jimmiemunyi]

Hello, Nice blog Might you know if it works on different tasks, e.g. Segmentation or object detection for the downstream task?