BYOL momentum update function is defined, but not used in the training script

[yuanenming]

https://github.com/PatrickHua/SimSiam/blob/fca181ed52274088fb39b92f3a9a57f97858e5d3/models/byol.py#L84

I am trying to apply different contrastive learning algorithms to some other fields. I found your implementation is really elegant and useful. However, I found that the BYOL model will not call the momentum update function.

So, is it a BUG, or I have missed it?

[PatrickHua]

I know ... I just can't figure out a way to fit this line of code in the training.

[triangleCZH]

@PatrickHua May I know about your concerns on the updating?

[joshnroy]

Bump on this. Is the function well defined but not implemented? It seems to me that you could put it in main.py but I'm not sure if that would mess anything up

[geekbgm]

anyone who solve this problem?

[liuzuo-byte]

I'm also having this problem, there is a problem calling this function, what should I do, has anyone solved it

[Taoww21480]

Yes, I also encountered this issue. During the training process of BYOL, when I trained several epochs, its accuracy remained zero, which is different from my training of SimSiam. SimSiam achieved an accuracy of over 30% at the beginning of the training, and I also found that you did not call update_moving_average() function in your byol.py file, so I referred to other people's code and called it in main.py. The calling code is as follows:

If args. model. name=="byol": model.module.update moving average()

Add the original the parameters of the update_moving_average() function were set in advance, and the momentum update parameter was set to 0.996. Therefore, I rerun it several times, but after training several epochs, the accuracy remained zero. So, I'm not sure how to solve this problem.

[Taoww21480]

I got it, during the training process, after several epochs were trained, the accuracy was 0. The reason for this is that the testing used here is the knn method, which needs to be set parameter “knn_monitor:True” in the "./configs/byol_cifar10.yaml“ file in order to perform knn testing and avoid the phenomenon of an accuracy of 0. After the byol training is completed, a linear evaluation will be conducted, and this evaluation will not show an accuracy of 0. At first, I thought the byol model had collapsed (but in reality, the training loss was gradually decreasing).

setting “knn_monitor:True” in the "./configs/byol_cifar10.yaml“ file can solve this issue.

Therefore, the main issue is still the setting the parameter ”knn_monitor:True“ in the "./configs/byol_cifar10.yaml“ file The monitor parameter is true only.

[Gitnew1]

Yes, I also encountered this issue. During the training process of BYOL, when I trained several epochs, its accuracy remained zero, which is different from my training of SimSiam. SimSiam achieved an accuracy of over 30% at the beginning of the training, and I also found that you did not call update_moving_average() function in your byol.py file, so I referred to other people's code and called it in main.py. The calling code is as follows:

If args. model. name=="byol": model.module.update moving average()

Add the original the parameters of the update_moving_average() function were set in advance, and the momentum update parameter was set to 0.996. Therefore, I rerun it several times, but after training several epochs, the accuracy remained zero. So, I'm not sure how to solve this problem.

Could you tell me where you add the calling code?