Better support for self supervised learning

[PulkitMishra]

🚀 Feature

Add support for self supervised learning, especially for multimodal input and improve the current vissl backed features.

Motivation

I have been working on multimodal self supervised learning for a while and have explored works like MMV, VATT, etc. The lack of a framework in the open source ecosystem for self supervised learning in general and for multimodal self supervised learning in particular is a tad disappointing.

Pitch

Torch Lightning ecosystem is amazing and it does a great job at increasing the speed with which researchers can try out different ideas. Since self supervised learning is a super trendy topic for DL researchers it would be great to have better support for the same.

Alternatives

I have looked into vissl, solo-learn and lightly however these libraries only provided (limited) support for self supervised learning on images and do not support videos or multimodal input. Further as discussed with @krshrimali on slack vissl(which flash relies on) does not seem to be actively maintained.

Additional context

Related slack discussion

[PulkitMishra]

hey @Borda @krshrimali i've cooked up something similar to MMV here : https://github.com/PulkitMishra/self-supervised-learning-exp/ but instead of MIL-NCE loss I'm simply using triplet loss as of now. Not sure how relevant it is to flash or useful for lightning in general. I plan to work more on it. Let me know if this can be useful to lightning somehow.

Also, I would appreciate if you or someone could maybe take a look at the code and suggest how to further improve it.

[krshrimali]

Hi, @PulkitMishra - Apologies for the delay in the response. We all were very occupied for the Lightning-AI Launch (https://lightning.ai/ ⚡) - and hence the delay. We are catching up with all the issues, and progress in Lightning Flash, and will get back to you on this as soon as possible. :)

[ethanwharris]

Hey @PulkitMishra thanks for the suggestions! It would be great to have better support for SSL methods in our ecosystem. One option would be to work on the (currently deprecated) SSL code we have in lightning-bolts to generalise it and make it more fit for use in a multi-modal setting. Let us know if that's something you'd be interested in :smiley:

cc @otaj who may have some thoughts here

[PulkitMishra]

Hi @krshrimali @ethanwharris first of all congrats on the launch. I have updated https://github.com/PulkitMishra/self-supervised-learning-exp/ and would love to hear your thoughts on the same.

I would also absolutely love to contribute to flash, bolts or anywhere in the lightning ecosystem and extend support for SSL. However, I am a bit new to the ecosystem so might need some help on that front.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Atharva-Phatak]

I am sorry to barge in here. But I believe flash uses visuals as the backend right? Since you mentioned lightning bolts could it be possible that slowly by slowly we start adding SSL models to lightning bolts and then use it as a backend for flash. Ofcourse this is a long-term game and will require a huge effort from everyone. Let me know what you think.