search

HHHedo / IBMIL

CVPR 2023 Highlight
59 stars 9 forks source link

failed to reproduce the results under (ResNet-18, DSMIL, Camelyon16) #20

Closed wuxchcandoit closed 3 months ago

wuxchcandoit commented 3 months ago

Excuse me. May I ask you a simple question? Is the classification task for dataset TCGA-NSCLC still a binary classification task even though TCGA-NSCLC includes two subtype?

wuxchcandoit commented 3 months ago

Besides, I failed to reproduce the results in the paper with the setting (ResNet-18 pretrained with ImageNet, DSMIL (without IBMIL), CAMELYON16) even though the public processed features in BaiduCloud (Camelyon16_ImageNet) have been used. I can only obtain about 73.xx AUC scores. What should I do to reproduce the results in the paper now? It that reasonable? Do you reproduce the same results as mine?

HHHedo commented 3 months ago

  1. The num_class for tcga is set to 2.

  2. The unstable training of dsmil is also observed by myself (see issue ) and others (see issue). One possible reason is that the attention-based aggregator failed to learn meaningful weight. You could observe these attention score during training and try different settings .

wuxchcandoit commented 3 months ago

Actually, I also failed to reproduce the results under the settings (ResNet-18 pretrained with ImageNet, TransMIL, Camelyon16). Therefore, the problem is not only about the unstable training of DSMIL. Are the public pre-computed features in BaiduCloud reliable?

HHHedo commented 3 months ago

Since all the aggregators used in our paper are open-resourced, you may reproduce the baselines based on their repositories.