Applying this Model to CIFAR10 Dataset

[KANG-Xudong]

Thank you so much for your attention.

I am trying to applying this model to some smaller size datasets like CIFAR-10 and MNIST. But during my training, I didn't find significant "learning" using this unsupervised method (I printed the NMI values between the pseudo labels and the true labels in each epoch, but find these values all below 0.1 in all 200 epoches, and no significant improvement was found).

I trained CIFAR-10 dataset with clusters number k = 10. Is this because this k values is too small? I also applied multiple times of training with learning rate lr = 0.001, 0.005, 0.01, 0.05, while all these results were not satisfactory.

I am wondering whether there are already experiment results applying this model on CIFAR-10 dataset, and I will be very appreciate if you might share your results with me or help me to implement.

I am a university student and I really need your help to finish my assignment.

[pvskand]

@KANG-Xudong, which network are you using for the training? Is it your custom network or are you training CIFAR-10 on existing net?

[KANG-Xudong]

@KANG-Xudong, which network are you using for the training? Is it your custom network or are you training CIFAR-10 on existing net?

I simply applied the AlexNet model in the /model/alexnet.py

[pvskand]

I guess that might be the problem. In case you are using Alexnet or VGG then it would not result in learning of good representations. One of the major reasons being the image size. CIFAR-10 has image of size 32x32 and the above mentioned networks take in 224x224 as input. So you might define a smaller network for CIFAR-10 and then go ahead with the learning.

[KANG-Xudong]

I guess that might be the problem. In case you are using Alexnet or VGG then it would not result in learning of good representations. One of the major reasons being the image size. CIFAR-10 has image of size 32x32 and the above mentioned networks take in 224x224 as input. So you might define a smaller network for CIFAR-10 and then go ahead with the learning.

Thank you for your answer. I have tried to resize the input image to the shape 244 x 244 to fit the network. Moreover, I have already tried to use this method (using the /models/alexnet.py as model) in supervised training, and get 81% accuracy with CIFAR-10 dataset. Thus, I think that may not result from the image size.

[pvskand]

You said you were using k=10, but it need not be the case that the number of classes in your dataset would directly correspond to the number of clusters. You might need to try some ablations with the value of k. Also when you use k=10, how many points are going in each cluster? Can you type in the stats here?

[mathildecaron31]

Hi, I have tried DeepCluster on smaller datasets such as CIFAR10. Actually the method doesn't seem to work well in this setting. I experimented with a Network In Network architecture. With a careful cross validation of parameters (k=100) and whitening the images in input I obtained decent performance but way above the performance reported by RotNet (https://arxiv.org/abs/1803.07728). For example, when training a SVM on top of block 2 of a NIN3 architecture I got 73% accuracy (RotNet: 82%; Random: 61%). Hope it helps

[hxixixh]

@mathildecaron31 Hi, I am working on a similar problem. Could you please provide more training information. Thanks!!

[Chen-Song]

Hi, I have tried DeepCluster on smaller datasets such as CIFAR10. Actually the method doesn't seem to work well in this setting. I experimented with a Network In Network architecture. With a careful cross validation of parameters (k=100) and whitening the images in input I obtained decent performance but way above the performance reported by RotNet (https://arxiv.org/abs/1803.07728). For example, when training a SVM on top of block 2 of a NIN3 architecture I got 73% accuracy (RotNet: 82%; Random: 61%). Hope it helps

Hi, Can you explain why this happens on small data sets? I also used the NIN architecture on CIFAR-10, but only got 7% higher classification results than random networks.

[chrysantd]

So, is there a conclusion about the effect on CIFAR10 dataset? I tried this method with several models like alexnet and resnet, but none of them show good results.

[Hzzone]

I have achieved 46.18 on cifar10, resnet18, 400 epochs, 100 clusters (the first conv is 3x3 and modified moco code, knn monitor), updated kmeans every 20 epochs. I do not think it is ok compared to moco ~90. Besides, the accuracy tends to gradually decrease...

[wufeim]

I just want to add some extra experimental results to this thread. I've experimented on CIFAR-10 with VGG-11 and LeNet. I experimented with 50/100/100 clusters with pca-downampled to 32/64/128, and various training strategies. This method doesn't seem to adapt well. In the best setting (fine-tuning without fixing conv base), there are 2 percent drop in performance. I like the idea of DeepCluster very much but I really cannot figure out why DeepCluster wouldn't work in my case.

[Ehtii]

The approach of deep cluster does not adapt to Cifar-10 dataset. I tried by resizing the images to match the alexnet/vgg16 input size, but model does not learn anything. The debugging showed that there are some nan/zero predictions from the model during the differerent batches of input data. This halts the loss and does not decrease. Conclusion: Deepcluster does not adapt to Cifar10.