search

vturrisi / solo-learn

solo-learn: a library of self-supervised methods for visual representation learning powered by Pytorch Lightning
MIT License
1.42k stars 185 forks source link

FFCV status #298

Closed priyamdey closed 2 years ago

priyamdey commented 2 years ago

Hi there,

I had noticed earlier that you guys had a FFCV branch for ffcv transforms support. But it's not there anymore. I was curious if you were able to get the 2 loader stuff (zipping them as was discussed here) working and were you able to test out the load timings for multiple crops. Any pointers would be helpful!

vturrisi commented 2 years ago

Hey,

I think I won't be if much help. I tried for a bit but it was pretty slow and needed a proper fix and not just a workaround. Also, FFCV doesn't support most of the transformation that we would require and reimplementing it would take a major effort. So, I ended up just dropping trying to add support for it. Seems that their project hasn't received any major updates for quite a while now, but if they fix the first thing I might give it another try.

priyamdey commented 2 years ago

Okay cool. Thanks for the update.

realliyifei commented 1 year ago

We are also working on adapting FFCV to SSL such as MoCo v2 to spend the training. By the information in this issue and the linked issue in ffcv repo, I think the conclusion from solo-learn authors is that it is impossible to adapt FFCV to MoCo v2 effectively? 

If yes, may I ask what is the fastest implementation of MoCo v2 to your best knowledge? Thank you!

vturrisi commented 1 year ago

Hey! I haven't spent enough time to dig deep and see how to adapt ffcv for self-sup methods, but it was not trivial when I tried. I would say the fastest (dataloading method) is Nvidia's dali or this tensorclass. Haven't tried the latter, but looks very promising.

realliyifei commented 1 year ago

Thanks! Very helpful! The Nvidia's dali is used in this repo (solo-learn), including in MoCo v2, so that means I can just use this solo library to speed the training of MoCo v2, right? Is there any document to indicate how fast MoCo v2 is improved here (sorry I can't find one via neither the README nor the paper)? Also, solo library vs tensorclass, which one is more promising to speed MoCo v2?

vturrisi commented 1 year ago

You can check the speed gains (in the described setting) here. You should expect a similar speedup by using dali, if you have a data loading bottleneck (smaller models will suffer more from this). solo-learn provides multiple self-supervised methods, with different dataloading schemes, whereas tensorclass is just a dataloader. They are not mutually exclusive, in fact, we plan on supporting it in the near future and see how it compares with other dataloading strategies. If you want to use tensorclass for now, you can simply add it as a dataloader to solo.

realliyifei commented 1 year ago

Got it! Thanks!