search

lucidrains / pixel-level-contrastive-learning

Implementation of Pixel-level Contrastive Learning, proposed in the paper "Propagate Yourself", in Pytorch
MIT License
252 stars 27 forks source link

Alpha - Combination of PixPro loss and PixCL #4

Closed WilhelmT closed 3 years ago

WilhelmT commented 3 years ago

Hi,

Thanks for your implementation. In the original paper the total loss is defined as L_PixPro + alphaL_inst (the instance level contrastive loss), while in your code it's L_PixPro + alphaL_PixCL (Pixel level contrastive loss). Any particular reason for why this is?

lucidrains commented 3 years ago

@WilhelmT Oh darn, I misunderstood the paper! Thanks for catching this

Looking at their L_inst though, I think I can make it even better by going for the BYOL approach rather than SimCLR (combining it with L_PixPro).

It's becoming apparent that negative samples are not needed https://arxiv.org/abs/2011.10566

lucidrains commented 3 years ago

@WilhelmT It also seems like for the instance and pixel representations come out at different levels in the Resnet, projected separately (I think) This may take a bit of work to make into a flexible framework

lucidrains commented 3 years ago

@WilhelmT I've combined this with BYOL instance level learning https://github.com/lucidrains/pixel-level-contrastive-learning/commit/59915ed70db80686aafca146f75e442fdbdd083e which should be superior to SimCLR, let me know if you spot any errors! Thank you for catching this!

WilhelmT commented 3 years ago

@lucidrains Nice. It would make sense if the combination with BYOL is superior, I wonder why the original authors did not do it.

Any plans to release pre-trained models when the implementation is finished if I may ask?

lucidrains commented 3 years ago

@WilhelmT yup, I intend to circle back and run it against BYOL alone to see how much higher the gains are. maybe once this is resolved as well https://github.com/lucidrains/pixel-level-contrastive-learning/issues/5

vasl12 commented 3 years ago

Hello, although the issue was closed, in the code the alpha issue is not corrected loss = pix_loss * self.alpha +instance_loss