search

IgorSusmelj / barlowtwins

Implementation of Barlow Twins paper
MIT License
100 stars 12 forks source link

Projection_mlp #4

Open Mushtaqml opened 3 years ago

Mushtaqml commented 3 years ago

Hi,

Are you using a projection network with following dimensions resnet (output) -> 512 -> 2048 -> 2048 ?

If that is the case then I am curious to know that why you decided to do it like this as in the paper they are having projection layer of 8192 dimensions.

Thanks

IgorSusmelj commented 3 years ago

We did some quick experiments and it seems that Barlow Twins works well with a similar model configuration as SimSiam. We essentially ended up using the SimSiam model just replacing the loss with Barlow Twins. We also tried increasing the last layer to 8192 dim output but the accuracy was pretty bad in my experiment. Maybe I didn't train long enough. I also have to admit that I didn't tune any hyperparameter. Getting ~90% on cifar10 test set with kNN is pretty good and was sufficient for me. I'll do more benchmarking once I have time. (vary the batch size, train for more epochs, model params...)