search

facebookresearch / simsiam

PyTorch implementation of SimSiam https//arxiv.org/abs/2011.10566
Other
1.15k stars 176 forks source link

Reproducing CIFAR-10 Results #9

Open ardywibowo opened 3 years ago

ardywibowo commented 3 years ago

Hello, I was trying to modify the code to work on the CIFAR-10 dataset and I'm having some trouble. I have changed the following:

  1. LR = 0.03
  2. Weight Decay = 0.0005
  3. Momentum = 0.9
  4. Batch Size = 512
  5. Removed GaussianBlur and Cropping
  6. Removed the 2nd set of Linear, BatchNorm, and ReLU layers in the projector.
  7. Modified the normalization parameters to CIFAR-10 statistics.
  8. Changed architecture to resnet18. The paper mentions ResNet18 for CIFAR, but I wasn't sure if it's different from resnet18.

But I can't seem to get it to work. I'm getting only about 37% accuracy with both the Linear finetuning and kNN. Do you have any additional implementation details that I may have missed?

ardywibowo commented 3 years ago

Update: Changed the augmentations to include resized cropping to a 32x32 image size and it improved the accuracy. Will update with new results once it finishes training.

Update 2: Training has just finished and I was able to get a decent 84% accuracy with kNN. I'm still wondering if I'm missing any additional details.

elias-ramzi commented 3 years ago

Hi, could you share the resized cropping you used ? Thanks !

ardywibowo commented 3 years ago

Sure, It's the same as the one done in the original code: transforms.RandomResizedCrop(32, scale=(0.2, 1.)).

endernewton commented 3 years ago

For details, it may be helpful to check the demo released in the moco codebase: https://colab.research.google.com/github/facebookresearch/moco/blob/colab-notebook/colab/moco_cifar10_demo.ipynb

elias-ramzi commented 3 years ago

Hi again,

With the CIFAR ResNet18 I am getting 88.4% on knn accuracy. What were your results (on the supplementary there is not the exact number for the knn evaluation) ?

I am not sure on several things (might be details): What is the size you used for the projection head and the predictor ? In the MoCo demo they use the default arguments for RandomResizeCrop what were your values ? Also could there be a difference coming from the fact that I am working on a single GPU (and not using the trick used in the MoCo demo to split the batches) ?

Thanks :)

ardywibowo commented 3 years ago

@elias-ramzi What did you do with your implementation? I'm only getting 84%. Did you copy the MoCo demo augmentation?

elias-ramzi commented 3 years ago

No I was use using the same augmentations as SimSiam without gaussian blur. Did you use the ResNet version of the MoCo demo (changed conv1, and remove maxpool) ?

ardywibowo commented 3 years ago

I see. No I didn't. Thanks! Let me try that.

Did you use 128 as the feature_dim or the default 2048? If you changed it to 128, did you change the pred_dim as well to 128/4 = 32?

elias-ramzi commented 3 years ago

In the supplementary material it is said that they used 2048, so I used the default settings for the projection head (with only two layers).

bhneo commented 2 years ago

Hi again,

With the CIFAR ResNet18 I am getting 88.4% on knn accuracy. What were your results (on the supplementary there is not the exact number for the knn evaluation) ?

I am not sure on several things (might be details): What is the size you used for the projection head and the predictor ? In the MoCo demo they use the default arguments for RandomResizeCrop what were your values ? Also could there be a difference coming from the fact that I am working on a single GPU (and not using the trick used in the MoCo demo to split the batches) ?

Thanks :)

Hi bro, how many parameters in your CIFAR ResNet18, I can get only 78% with CIFAR resnet 18, but 89% with resnet 18

elias-ramzi commented 2 years ago

Hi, sorry for the late response:

I have 14580800 parameters for the CIFAR resnet18

meobach commented 2 years ago

hi @ardywibowo , have you tried it with tiny-imagenet dataset, I have trained and used the pre-trained model to do an linear classification but result look really bad Capture

liuzuo-byte commented 1 year ago

我在高光谱数据上跑这个代码的时候，线性分类精度只有55%，请问各位大佬，我需要在哪些地方改进，我的数据增强就是单一的高斯噪声，我的代码有问题吗，还是说我要修改优化器还是啥

您好，我正在尝试修改代码以在CIFAR-10数据集上工作，我遇到了一些麻烦。我更改了以下内容：

  1. LR = 0.03
  2. 重量衰减 = 0.0005
  3. 动量 = 0.9
  4. 批大小 = 512
  5. 删除了高斯模糊和裁剪
  6. 已删除投影仪中的第二组线性、批处理规范和 ReLU 图层。
  7. 已将规范化参数修改为 CIFAR-10 统计信息。
  8. 已将体系结构更改为 resnet18。这篇论文提到了CIFAR的ResNet18，但我不确定它是否与resnet18不同。

但我似乎无法让它工作。线性微调和 kNN 的准确率只有 37% 左右。你们是否有任何我可能错过的其他实现细节？

你好，你后面是如何提高的线性分类精度呢

liuzuo-byte commented 1 year ago

我在高光谱数据上跑这个代码的时候，线性分类精度只有55%，请问各位大佬，我需要在哪些地方改进，我的数据增强就是单一的高斯噪声，我的代码有问题吗，还是说我要修改优化器还是啥