Open OlivierDehaene opened 3 years ago
I am available to work on this :)
Hi @OlivierDehaene , this sounds amazing! go ahead with the implementation and let us know if we can help in any way! :)
@prigoyal, @QuentinDuval,
There is something that puzzles me with the loss.
In SimCLR, we compare view1 with view2 but we also compare each view with itself and mask the diagonal. This is done to add negative pairs. Here, since view1 and view2 are not sampled from the same data augmentation (NNCLR uses the BYOL pipeline where view1 and view2 have different gaussian blur and solarization probabilities), it is not clear if we are allowed to do that. If it is not allowed, it effectively divides the batch size by 2 (EDIT: it divides the number of negative pairs by 2).
From the pseudocode above, we can infer that we should only compare view1 and view2. However, it is not clear if they omitted it to make the pseudocode more explicit or because it has an effect on performance.
TLDR: should we only compare views that are sampled from different data augmentations?
@prigoyal, @QuentinDuval,
There is something that puzzles me with the loss.
In SimCLR, we compare view1 with view2 but we also compare each view with itself and mask the diagonal. This is done to add negative pairs. Here, since view1 and view2 are not sampled from the same data augmentation (NNCLR uses the BYOL pipeline where view1 and view2 have different gaussian blur and solarization probabilities), it is not clear if we are allowed to do that. If it is not allowed, it effectively divides the batch size by 2 (EDIT: it divides the number of negative pairs by 2).
From the pseudocode above, we can infer that we should only compare view1 and view2. However, it is not clear if they omitted it to make the pseudocode more explicit or because it has an effect on performance.
TLDR: should we only compare views that are sampled from different data augmentations?
This is a good question (hard to answer because we do not have the implementation yet) so I went back to the paper, and the loss is defined as:
They mention right after that they make it symmetric by adding the following term as well (but they also say that this has little influence on the performance):
So my understanding is that we should have the same number of negative pairs as SimCLR.
Hello~ May i ask a question about the implementation of NNCLR in here ? Although the torch.select_index may cover the detail of implementation, I wonder that such operation is differentiable ?
In the NNCLR, the enoder, whose embedding is replaced by the support set, will not be updated in that step. So, it's means we will deteach the embedding before feed it into the query method of support set ?
Thanks for any suggestion..
Please code is available?
Please code is available?
you can move to solo-learn, the more powerful SSL framework with readable repository. π https://github.com/vturrisi/solo-learn
and say goodnight to vissl ~
Please code is available?
Please code is available?
you can move to solo-learn, the more powerful SSL framework with readable repository. π https://github.com/vturrisi/solo-learn
and say goodnight to vissl ~
Please code is available?
Thanks for your kind reply !
π New SSL approach addition
Approach description
NNCLR: https://arxiv.org/abs/2104.14548
Architecture
Pseudocode
Open source status