Is the bias term in the predictor/projector in the Linear layer necessary?

taoyang1122 / pytorch-SimSiam

A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Apache License 2.0

78 stars 8 forks source link

Is the bias term in the predictor/projector in the Linear layer necessary? #8

Open flishwang opened 3 years ago

flishwang commented 3 years ago

Hi, thanks for your work. It's wonderful. But I noticed that the bias arg in all nn.Linear layer in models/simsiam.py is set as its default value True. To my knowledge, the bias in linear/conv before batch_norm will take no effect as BN will remove all biases. So I'd like to know whether the bias term in Linear before batch_norm could be removed during training.