search

facebookresearch / deepcluster

Deep Clustering for Unsupervised Learning of Visual Features
Other
1.69k stars 325 forks source link

ValueError: 'a' cannot be empty unless no samples are taken #43

Closed dswf65411 closed 5 years ago

dswf65411 commented 5 years ago

The error occurs at 32nd epoch and run normally before it, I guess maybe Kmeans has assigned an empty cluster so a list become empty?

package version:

main.py:265: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
  losses.update(loss.data[0], input_tensor.size(0))
Epoch: [32][0/170]  Time: 0.575 (0.575) Data: 0.561 (0.561) Loss: 4.6828 (4.6828)
Epoch: [32][10/170] Time: 0.360 (0.376) Data: 0.000 (0.082) Loss: 3.0669 (3.4403)
Epoch: [32][20/170] Time: 0.359 (0.368) Data: 0.000 (0.060) Loss: 3.7505 (3.2823)
Epoch: [32][30/170] Time: 0.358 (0.365) Data: 0.000 (0.052) Loss: 3.1550 (3.1970)
Epoch: [32][40/170] Time: 0.358 (0.363) Data: 0.000 (0.048) Loss: 3.0475 (3.0846)
Epoch: [32][50/170] Time: 0.358 (0.362) Data: 0.000 (0.045) Loss: 3.0932 (2.9922)
Epoch: [32][60/170] Time: 0.359 (0.361) Data: 0.000 (0.044) Loss: 2.8730 (3.0125)
Epoch: [32][70/170] Time: 0.359 (0.361) Data: 0.000 (0.043) Loss: 3.4860 (2.9648)
Epoch: [32][80/170] Time: 0.358 (0.361) Data: 0.000 (0.042) Loss: 3.1373 (2.9436)
Epoch: [32][90/170] Time: 0.359 (0.361) Data: 0.000 (0.041) Loss: 2.6147 (2.9145)
Epoch: [32][100/170]    Time: 0.359 (0.360) Data: 0.000 (0.040) Loss: 2.6068 (2.8890)
Epoch: [32][110/170]    Time: 0.360 (0.360) Data: 0.000 (0.040) Loss: 2.4559 (2.8653)
Epoch: [32][120/170]    Time: 0.359 (0.360) Data: 0.000 (0.040) Loss: 2.2136 (2.8452)
Epoch: [32][130/170]    Time: 0.358 (0.360) Data: 0.000 (0.039) Loss: 2.4173 (2.8292)
Epoch: [32][140/170]    Time: 0.358 (0.360) Data: 0.000 (0.039) Loss: 2.7725 (2.8246)
Epoch: [32][150/170]    Time: 0.360 (0.360) Data: 0.000 (0.039) Loss: 2.2551 (2.7988)
Epoch: [32][160/170]    Time: 0.358 (0.360) Data: 0.000 (0.038) Loss: 2.4294 (2.7789)
###### Epoch [32] ######
Time: 61.166 s
Clustering loss: 1608.053
ConvNet loss: 2.779
/home/yaochu/anaconda3/envs/env1/lib/python3.7/site-packages/sklearn/metrics/cluster/supervised.py:859: FutureWarning: The behavior of NMI will change in version 0.22. To match the behavior of 'v_measure_score', NMI will use average_method='arithmetic' by default.
  FutureWarning)
NMI against previous assignment: 0.723
#######################

Compute features
main.py:296: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  input_var = torch.autograd.Variable(input_tensor.cuda(), volatile=True)
0 / 170 Time: 0.777 (0.777)
10 / 170    Time: 0.282 (0.212)
20 / 170    Time: 0.121 (0.176)
30 / 170    Time: 0.111 (0.168)
40 / 170    Time: 0.396 (0.171)
50 / 170    Time: 0.112 (0.163)
60 / 170    Time: 0.208 (0.161)
70 / 170    Time: 0.111 (0.163)
80 / 170    Time: 0.110 (0.162)
90 / 170    Time: 0.112 (0.161)
100 / 170   Time: 0.111 (0.161)
110 / 170   Time: 0.111 (0.162)
120 / 170   Time: 0.110 (0.161)
130 / 170   Time: 0.109 (0.159)
140 / 170   Time: 0.107 (0.158)
150 / 170   Time: 0.151 (0.157)
160 / 170   Time: 0.110 (0.156)
WARNING clustering 3384 points to 100 centroids: please provide at least 3900 training points
k-means loss evolution: [3554.9004 1898.8888 1713.1171 1647.2383 1626.5762 1618.4319 1610.1742
 1604.1263 1599.3331 1596.9933 1595.8425 1594.5605 1593.4675 1592.6006
 1592.2092 1591.5988 1591.486  1591.486  1591.486  1591.486 ]
k-means time: 5 s
Traceback (most recent call last):
  File "main.py", line 320, in <module>
    main()
  File "main.py", line 160, in main
    deepcluster.images_lists)
  File "/home/yaochu/deepcluster/util.py", line 59, in __init__
    self.indexes = self.generate_indexes_epoch()
  File "/home/yaochu/deepcluster/util.py", line 69, in generate_indexes_epoch
    replace=(len(self.images_lists[i]) <= size_per_pseudolabel)
  File "mtrand.pyx", line 1125, in mtrand.RandomState.choice
ValueError: 'a' cannot be empty unless no samples are taken

Please help, thank you!

mathildecaron31 commented 5 years ago

Hi, Thank you so much for your interest and for raising this issue.

Reading the error, it seems that you actually have at least one cluster empty. I had always assumed that Faiss was dealing with empty clusters without actually checking where they do it in the code... I think I made a confusion with this piece of code from FastText.

So actually the method works without reassigning empty clusters, and I never discovered it because I've never had any.

Note that in the implementation of DeeperCluster I do reassign empty clusters with the following piece of code.

With this fix you shouldn't get the error.