Model evaluation

[Will0x6c5f]

Hello

First of all, thank you for publishing this code. I'm having difficulty in evaluating the trained model. Adopting eval.py form SCAN seems not straight forward and I'm not sure whether I've done it correctly. Having trained the model for 401 epoch, The ACC remains 0.1. Here is the training log from last epoch:

use time : 64.4565258026123 Epoch: [399][ 0/391] Time 33.955 (33.955) Data 33.852 (33.852) Loss 7.7962e+00 (7.7962e+00) Epoch: [399][100/391] Time 0.075 ( 0.415) Data 0.000 ( 0.336) Loss 7.5979e+00 (7.9200e+00) Epoch: [399][200/391] Time 0.075 ( 0.248) Data 0.000 ( 0.169) Loss 7.7325e+00 (7.8598e+00) Epoch: [399][300/391] Time 0.074 ( 0.191) Data 0.000 ( 0.113) Loss 7.3591e+00 (7.8692e+00) Epoch: [399][391/391] Time 0.058 ( 0.165) Data 0.000 ( 0.087) Loss 7.8860e+00 (7.8535e+00) max and min cluster size for 10-class clustering is (5120.0,4736.0) max and min cluster size for 20-class clustering is (2688.0,2304.0) max and min cluster size for 30-class clustering is (1920.0,1408.0) max and min cluster size for 40-class clustering is (1664.0,976.0) max and min cluster size for 50-class clustering is (1280.0,512.0) max and min cluster size for 60-class clustering is (1152.0,384.0) max and min cluster size for 70-class clustering is (1152.0,256.0) max and min cluster size for 80-class clustering is (896.0,256.0) max and min cluster size for 90-class clustering is (1152.0,0.0) max and min cluster size for 100-class clustering is (896.0,0.0) use time : 65.68861746788025 Epoch: [400][ 0/391] Time 32.947 (32.947) Data 32.842 (32.842) Loss 8.7333e+00 (8.7333e+00) Epoch: [400][100/391] Time 0.074 ( 0.404) Data 0.000 ( 0.326) Loss 7.6575e+00 (7.7840e+00) Epoch: [400][200/391] Time 0.092 ( 0.242) Data 0.000 ( 0.164) Loss 8.2370e+00 (7.8897e+00) Epoch: [400][300/391] Time 0.096 ( 0.188) Data 0.000 ( 0.110) Loss 7.3230e+00 (7.8503e+00) Epoch: [400][391/391] Time 0.059 ( 0.162) Data 0.000 ( 0.084) Loss 7.5377e+00 (7.8445e+00) max and min cluster size for 10-class clustering is (5248.0,4864.0) max and min cluster size for 20-class clustering is (2640.0,2304.0) max and min cluster size for 30-class clustering is (1792.0,1536.0) max and min cluster size for 40-class clustering is (1408.0,1024.0) max and min cluster size for 50-class clustering is (1408.0,640.0) max and min cluster size for 60-class clustering is (1280.0,384.0) max and min cluster size for 70-class clustering is (1024.0,256.0) max and min cluster size for 80-class clustering is (1024.0,256.0) max and min cluster size for 90-class clustering is (896.0,0.0) max and min cluster size for 100-class clustering is (768.0,128.0) use time : 64.5689332485199

Here is my adaptation to get the cluster prediction (in order to reuse code from SCAN) :

I'm not sure if this is the right way to get the prediction. Please correct me if I'm wrong. Surely it will be better if you can provide the evaluation script.

[qian-qi]

Hi,

The evaluation is conducted on predicted labels. So you have to obtain the pseudo label, i.e., argmax on logits, for each instance.

[shashankkotyan]

@Will0x6c5f Did you have any success in the Evaluation? I also get high loss after multiple epochs, similar to you and suspect the model is not learning...

@qian-qi Here are my logs for your reference. (I also print individual Loss Cluster: loss_c and Loss Projector: loss_x)

Epoch: [140][  0/391]   Time  2.971 ( 2.971)    Data  2.886 ( 2.886)    Loss 3.6432e+00 (3.6432e+00)    Loss Cluster 1.2673e+00 (1.2673e+00)Loss Projector 2.3760e+00 (2.3760e+00)
Epoch: [140][100/391]   Time  0.072 ( 0.102)    Data  0.000 ( 0.029)    Loss 3.6309e+00 (3.7898e+00)    Loss Cluster 1.2127e+00 (1.3217e+00)Loss Projector 2.4181e+00 (2.4681e+00)
Epoch: [140][200/391]   Time  0.074 ( 0.088)    Data  0.000 ( 0.014)    Loss 3.8096e+00 (3.7591e+00)    Loss Cluster 1.3023e+00 (1.2908e+00)Loss Projector 2.5072e+00 (2.4683e+00)
Epoch: [140][300/391]   Time  0.072 ( 0.083)    Data  0.000 ( 0.010)    Loss 3.6605e+00 (3.7354e+00)    Loss Cluster 1.1973e+00 (1.2694e+00)Loss Projector 2.4631e+00 (2.4660e+00)
Epoch: [140][391/391]   Time  0.059 ( 0.081)    Data  0.000 ( 0.007)    Loss 3.5559e+00 (3.7160e+00)    Loss Cluster 1.1448e+00 (1.2514e+00)Loss Projector 2.4110e+00 (2.4646e+00)
max and min cluster size for 10-class clustering is (5881.0,4600.0)
max and min cluster size for 20-class clustering is (3163.0,2280.0)
max and min cluster size for 30-class clustering is (2319.0,1481.0)
max and min cluster size for 40-class clustering is (1745.0,1127.0)
max and min cluster size for 50-class clustering is (1319.0,896.0)
max and min cluster size for 60-class clustering is (1290.0,740.0)
max and min cluster size for 70-class clustering is (963.0,628.0)
max and min cluster size for 80-class clustering is (881.0,543.0)
max and min cluster size for 90-class clustering is (828.0,485.0)
max and min cluster size for 100-class clustering is (741.0,441.0)
use time : 31.691771507263184