Request for Evaluation Code Release

[vegetable-lion]

Thank you for your outstanding work!

I tried to replicate your results, but I noticed that the evaluation code has not been released. Could you please consider releasing the evaluation code? Additionally, during evaluation, is the self.assign_labels from the trained SeCu model directly usable as the clustering assignments for evaluation?

[qian-qi]

Thank you for your interest. We adopt the same evaluation pipeline as SCAN and you can find the code there.

[vegetable-lion]

Thank you for your interest. We adopt the same evaluation pipeline as SCAN and you can find the code there.

Thank you so much for your quick response! I tried using SCAN’s evaluation method to reproduce the CIFAR-10 results for run_cifar10_entropy.sh but I’m getting very low accuracy. I’m not sure where things went wrong. I would greatly appreciate any guidance you could offer.

Below is the log from the last epoch of my training:

Epoch: [400][  0/391]    Time 18.610 (18.610)    Data 18.502 (18.502)    Loss 2.0493e+00 (2.0493e+00)
Epoch: [400][100/391]    Time  0.077 ( 0.261)    Data  0.000 ( 0.183)    Loss 2.0409e+00 (2.0670e+00)
Epoch: [400][200/391]    Time  0.079 ( 0.170)    Data  0.000 ( 0.092)    Loss 2.0763e+00 (2.0713e+00)
Epoch: [400][300/391]    Time  0.084 ( 0.139)    Data  0.000 ( 0.062)    Loss 2.2931e+00 (2.0720e+00)
Epoch: [400][391/391]    Time  0.067 ( 0.125)    Data  0.000 ( 0.048)    Loss 2.1085e+00 (2.0722e+00)
max and min cluster size for 10-class clustering is (5132.0,4634.0)
max and min cluster size for 20-class clustering is (3784.0,2070.0)
max and min cluster size for 30-class clustering is (2013.0,1388.0)
max and min cluster size for 40-class clustering is (1438.0,1080.0)
max and min cluster size for 50-class clustering is (1102.0,829.0)
max and min cluster size for 60-class clustering is (1046.0,695.0)
max and min cluster size for 70-class clustering is (951.0,599.0)
max and min cluster size for 80-class clustering is (733.0,515.0)
max and min cluster size for 90-class clustering is (688.0,467.0)
max and min cluster size for 100-class clustering is (672.0,411.0)
use time : 49.63020062446594

The prediction function I added in class SeCu is as follows:

class SeCu(nn.Module):
    ...
    @torch.no_grad()
    def get_pred(self,x):
        x1 = self.encoder(x)
        x1_proj = F.normalize(x1, dim=1)
        head_idx = 0
        cur_c = F.normalize(getattr(self, "center_" + str(head_idx)), dim=0)
        proj_c1 = x1_proj @ cur_c
        # print(proj_c1.shape)
        return proj_c1

The evaluation method is as follows:

def cluster_metric(label, pred):
    nmi = metrics.normalized_mutual_info_score(label, pred)
    ari = metrics.adjusted_rand_score(label, pred)
    pred_adjusted = get_y_preds(label, pred, len(set(label)))
    acc = metrics.accuracy_score(pred_adjusted, label)
    print(
        "[Clustering Result]: ACC = {:.2f}, NMI = {:.2f}, ARI = {:.2f}".format(
            acc * 100, nmi * 100, ari * 100
        )
    )

def calculate_cost_matrix(C, n_clusters):
    cost_matrix = np.zeros((n_clusters, n_clusters))
    # cost_matrix[i,j] will be the cost of assigning cluster i to label j
    for j in range(n_clusters):
        s = np.sum(C[:, j])  # number of examples in cluster i
        for i in range(n_clusters):
            t = C[i, j]
            cost_matrix[j, i] = s - t
    return cost_matrix

def get_cluster_labels_from_indices(indices):
    n_clusters = len(indices)
    cluster_labels = np.zeros(n_clusters)
    for i in range(n_clusters):
        cluster_labels[i] = indices[i][1]
    return cluster_labels

def get_y_preds(y_true, cluster_assignments, n_clusters):
    """
    Computes the predicted labels, where label assignments now
    correspond to the actual labels in y_true (as estimated by Munkres)
    cluster_assignments:    array of labels, outputted by kmeans
    y_true:                 true labels
    n_clusters:             number of clusters in the dataset
    returns:    a tuple containing the accuracy and confusion matrix,
                in that order
    """
    confusion_matrix = metrics.confusion_matrix(
        y_true, cluster_assignments, labels=None
    )
    # compute accuracy based on optimal 1:1 assignment of clusters to labels
    cost_matrix = calculate_cost_matrix(confusion_matrix, n_clusters)
    indices = Munkres().compute(cost_matrix)
    kmeans_to_true_cluster_labels = get_cluster_labels_from_indices(indices)

    if np.min(cluster_assignments) != 0:
        cluster_assignments = cluster_assignments - np.min(cluster_assignments)
    y_pred = kmeans_to_true_cluster_labels[cluster_assignments]
    return y_pred

def main():
    args = parser.parse_args()
    print(args)
    if args.seed is not None:
        random.seed(args.seed)
        torch.manual_seed(args.seed)
        cudnn.deterministic = True
        warnings.warn('You have chosen to seed training. '
                      'This will turn on the CUDNN deterministic setting, '
                      'which can slow down your training considerably! '
                      'You may see unexpected behavior when restarting '
                      'from checkpoints.')

    if args.gpu is not None:
        warnings.warn('You have chosen a specific GPU. This will completely '
                      'disable data parallelism.')

    if args.dist_url == "env://" and args.world_size == -1:
        args.world_size = int(os.environ["WORLD_SIZE"])

    args.distributed = args.world_size > 1 or args.multiprocessing_distributed

    ngpus_per_node = torch.cuda.device_count()
    if args.multiprocessing_distributed:
        # Since we have ngpus_per_node processes per node, the total world_size
        # needs to be adjusted accordingly
        args.world_size = ngpus_per_node * args.world_size
        # Use torch.multiprocessing.spawn to launch distributed processes: the
        # main_worker process function
        mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
    else:
        # Simply call main_worker function
        main_worker(args.gpu, ngpus_per_node, args)

def main_worker(gpu, ngpus_per_node, args):
    args.gpu = gpu

    # suppress printing if not master
    if args.multiprocessing_distributed and args.gpu != 0:
        def print_pass(*args):
            pass

        builtins.print = print_pass

    if args.gpu is not None:
        print("Use GPU: {} for training".format(args.gpu))

    if args.distributed:
        if args.dist_url == "env://" and args.rank == -1:
            args.rank = int(os.environ["RANK"])
        if args.multiprocessing_distributed:
            # For multiprocessing distributed training, rank needs to be the
            # global rank among all the processes
            args.rank = args.rank * ngpus_per_node + gpu
        dist.init_process_group(backend=args.dist_backend, init_method=args.dist_url,
                                world_size=args.world_size, rank=args.rank)
    # create model
    assert (len(args.secu_k) == args.secu_num_head)
    print("=> creating model")
    if args.data_name == 'stl10':
        from nets.resnet_stl import resnet18
    elif args.data_name=='cifar10' or args.data_name=='cifar100':
        from nets.resnet_cifar import resnet18
    else:
        print("Input data set is not supported")
        return
    model = secu.builder.SeCu(
        base_encoder=resnet18,
        K=args.secu_k,
        tx=args.secu_tx,
        tw=args.secu_tw,
        dim=args.secu_dim,
        num_ins=args.secu_num_ins,
        alpha=args.secu_alpha,
        dual_lr=args.secu_dual_lr,
        lratio=args.secu_lratio,
        constraint=args.secu_cst
    )
    model = nn.SyncBatchNorm.convert_sync_batchnorm(model)
    save_mlp = os.path.join('model', f"secu_entropy_{args.data_name}_0400.pth.tar")
    checkpoint = torch.load(save_mlp)
    new_state_dict = {}
    for k, v in checkpoint['state_dict'].items():
        if k.startswith('module.'):
            new_state_dict[k[7:]] = v  # Remove the 'module.' prefix
        else:
            new_state_dict[k] = v
    model.load_state_dict(new_state_dict)
    model.load_param()
    cudnn.benchmark = True
    model.cuda()
    model.eval()
    # Data loading code
    testdir = os.path.join(args.data, 'test')
    if args.data_name == 'cifar10':
        normalize = transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                                         std=[0.2023, 0.1994, 0.2010])
        crop_size = 32
    elif args.data_name == 'cifar100':
        normalize = transforms.Normalize(mean=[0.5071, 0.4867, 0.4408],
                                         std=[0.2675, 0.2565, 0.2761])
        crop_size = 32
    elif 'stl' in args.data_name:
        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
        crop_size = 96
    aug_ = [
        transforms.CenterCrop(crop_size),
        transforms.ToTensor(),
        normalize
    ]
    test_dataset = secu.folder.ImageFolder(
        testdir,
        secu.loader.SingleCropsTransform(transforms.Compose(aug_)))
    test_sampler = None
    test_loader = torch.utils.data.DataLoader(
        test_dataset, batch_size=args.batch_size, shuffle=(test_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=test_sampler, drop_last=False)

    predictions = []
    probs = []
    labels_test = []
    for i, (images, target) in enumerate(test_loader):
        images = images.cuda()
        output = model.get_pred(images)
        predictions.append(torch.argmax(output, dim=1))
        probs.append(F.softmax(output, dim=1))
        labels_test.append(target.cuda())
    predictions = torch.cat(predictions).cpu().numpy()
    probs = torch.cat(probs).cpu().numpy()
    labels_test = torch.cat(labels_test).cpu().numpy()
    print(predictions.shape)
    print(probs.shape)
    cluster_metric(labels_test, predictions)

The result is

[Clustering Result]: ACC = 0.10, NMI = 39.98, ARI = 0.00

[qian-qi]

Thank you for your efforts. Please note that we have secu.folder.ImageFolder for unsupervised training, which assigns the unique id for each instance rather than ground-truth labels. The standard torchvision.datasets.ImageFolder should be used for evaluation that returns target labels.

test_dataset = torchvision.datasets.ImageFolder(testdir, transform=transforms.Compose([transforms.ToTensor(),normalize]))
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=args.batch_size, shuffle=False, num_workers=args.workers, pin_memory=True)

[vegetable-lion]

torchvision.datasets.ImageFolder

I didn’t notice this difference... Thank you so much for pointing it out! After making the changes, I successfully reproduced the results.

ACC = 87.83, NMI = 78.79, ARI = 76.94

[LYJhere]

Thank you, I got it.