Reproducibility of the experiment

[miziha-zp]

Dear Authors: I have followed the instructions in README and run your code on the environment provided in your requirement.yaml for three times. However, the results I got is only (0.7952, 0.7702, 0.7705,) in ACC(cifar-10), which is far from your reported results. Can you help me with the reproducibility of the experiment? Thanks

[ZhiyuanDang]

Hi @miziha-zp ,

Have you solved your PyTorch version issue? I will rerun the code to reproduce the results (At now, in Epoch 9/100, we get 0.81 ACC in CIFAR-10). And, have you used the simclr.py provided by us to generate the correct nearest neighbors?

Meanwhile, could you provide some logs help me to locate the problem?

[miziha-zp]

thanks for your reply, rerun your code on a 2080Ti the pytorch version issue was gone, and I did use simclr.py to generate the correct nearest neighbors, ohterwise, the scan.py won't run. one of my experiment log is here

[ZhiyuanDang]

My first epoch result is 0.5689 higher than your results 0.4981. Maybe something wrong in simclr.py, I consider uploading the model and took neighbors to fix this issue.

Or could you please provide the output log of simclr.py?

[miziha-zp]

thanks for your patience, do you mean that you using another pretrained model? can you please provide the pretrain scirpt for my experiment? all my experiment now is using the pretrained model download from scan repo as your instructions in README the output log of simclr.py is below

python simclr.py --config_env configs/env.yml --config_exp configs/pretext/simclr_cifar10.yml
{'setup': 'simclr', 'backbone': 'resnet18', 'model_kwargs': {'head': 'mlp', 'features_dim': 128}, 'train_db_name': 'cifar-10', 'val_db_name': 'cifar-10', 'num_classes': 10, 'criterion': 'simclr', 'criterion_kwargs': {'temperature': 0.1}, 'epochs': 500, 'optimizer': 'sgd', 'optimizer_kwargs': {'nesterov': False, 'weight_decay': 0.0001, 'momentum': 0.9, 'lr': 0.4}, 'scheduler': 'cosine', 'scheduler_kwargs': {'lr_decay_rate': 0.1}, 'batch_size': 512, 'num_workers': 8, 'augmentation_strategy': 'simclr', 'augmentation_kwargs': {'random_resized_crop': {'size': 32, 'scale': [0.2, 1.0]}, 'color_jitter_random_apply': {'p': 0.8}, 'color_jitter': {'brightness': 0.4, 'contrast': 0.4, 'saturation': 0.4, 'hue': 0.1}, 'random_grayscale': {'p': 0.2}, 'normalize': {'mean': [0.4914, 0.4822, 0.4465], 'std': [0.2023, 0.1994, 0.201]}}, 'transformation_kwargs': {'crop_size': 32, 'normalize': {'mean': [0.4914, 0.4822, 0.4465], 'std': [0.2023, 0.1994, 0.201]}}, 'pretext_dir': './results/cifar-10/pretext', 'pretext_checkpoint': './results/cifar-10/pretext/checkpoint.pth.tar', 'pretext_model': './results/cifar-10/pretext/model.pth.tar', 'topk_neighbors_train_path': './results/cifar-10/pretext/topk-train-neighbors.npy', 'topk_neighbors_val_path': './results/cifar-10/pretext/topk-val-neighbors.npy'}
Retrieve model
Model is ContrastiveModel
Model parameters: 11.50M
Set CuDNN benchmark
Retrieve dataset
Train transforms: Compose(
    RandomResizedCrop(size=(32, 32), scale=(0.2, 1.0), ratio=(0.75, 1.3333), interpolation=PIL.Image.BILINEAR)
    RandomHorizontalFlip(p=0.5)
    RandomApply(
    p=0.8
    ColorJitter(brightness=[0.6, 1.4], contrast=[0.6, 1.4], saturation=[0.6, 1.4], hue=[-0.1, 0.1])
)
    RandomGrayscale(p=0.2)
    ToTensor()
    Normalize(mean=[0.4914, 0.4822, 0.4465], std=[0.2023, 0.1994, 0.201])
)
Validation transforms: Compose(
    CenterCrop(size=(32, 32))
    ToTensor()
    Normalize(mean=[0.4914, 0.4822, 0.4465], std=[0.2023, 0.1994, 0.201])
)
Files already downloaded and verified
Files already downloaded and verified
Dataset contains 50000/10000 train/val samples
Build MemoryBank
Files already downloaded and verified
Retrieve criterion
Criterion is SimCLRLoss
Retrieve optimizer
SGD (
Parameter Group 0
    dampening: 0
    lr: 0.4
    momentum: 0.9
    nesterov: False
    weight_decay: 0.0001
)
Starting main loop
Fill memory bank for mining the nearest neighbors (train) ...
Fill Memory Bank [0/98]
Mine the nearest neighbors (Top-20)
Accuracy of top-20 nearest neighbors on train set is 83.90
Fill memory bank for mining the nearest neighbors (val) ...
Fill Memory Bank [0/20]
Mine the nearest neighbors (Top-5)
Accuracy of top-5 nearest neighbors on val set is 81.87

[ZhiyuanDang]

No, I always use the pre-trained model from SCAN. The output log is the same as mine. Therefore, the simclr.py I uploaded is correct.

Since I run the code in multiple machines, it is possible I upload an old version, I will check the code.

[miziha-zp]

thanks for your work.