SimCLR loss for self-supervised speech representation learning

[monatis]

Hi, Self-supervised pretraining for speech representation is a promising technique for developing ASR in resource-constraint languages with little transcribed data, and SimCLR is applied with success for this purpose in recent papers. Any plan to add support for SimCLR contrastive objective to this package?

Relevant papers

• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

[nglehuy]

@monatis Feel free to contribute any new approaches to this repo 😄 I have many plans, but lack of time to implement. Therefore it would be great if anyone can help this repo grow. (Don't worry, I'm still active for quite a long time, but new methods take time to arrive if only me is the one who implements them).

[monatis]

@usimarit great! I'm trying to fully understand how the whole package comes together to implement the method in the simplest way. PR will come when it's ready. I believe that it'll help add support for many more languages with little transcribed data. Personally, I prepared a large audio-only speech dataset for self-supervised pretraining in Turkish which has only 20-hour transcribed data in the public domain.