How to Manually Selecting Negative Samples

[bryanbocao]

Thanks for the awesome framework!

It seems that negative samples are by default the ones from a different class, is there a way to manually choose negative samples?

For instance, in a video, I would like to select a frame at time t=ti and thus the augmentations of this frame are positive examples. Then I would like to choose frames from another timestamp t!=ti as negative samples.

Thanks!

[philippmwirth]

Hi @bryanbocao

Most contrastive learning frameworks consider all other samples from the batch as negatives. If you want to implement extra logic for distinguishing positives from negatives I'd suggest to look at torch's triplet loss or to extend one of our contrastive loss functions with custom logic.

If you're interested in self-supervised learning on video data you should also check out CCL (cycle-contrast) and CycleCL although we don't have support for these methods yet.

[guarin]

If you want to use the classical InfoNCE or NTXent loss you can adapt our implementation, the only thing you have to change is how you create the mask that defines the positive/negative pairs (diag_mask):

https://github.com/lightly-ai/lightly/blob/66ad1b40ebf3b53512703c774988211ce283211f/lightly/loss/ntx_ent_loss.py#L148-L150

It might be easiest if you just pass the mask as an extra parameter to the forward method of the loss.

[guarin]

I'll close this issue, please reopen if there are more questions :)