search

xmed-lab / DHC

MICCAI 2023: DHC: Dual-debiased Heterogeneous Co-training Framework for Class-imbalanced Semi-supervised Medical Image Segmentation
MIT License
51 stars 3 forks source link

Model training performance does not improve. #6

Closed emma000730 closed 8 months ago

emma000730 commented 8 months ago

Hello! I downloaded your preprocessed Synapse dataset and trained according to the instructions, but the training results did not improve. What could be the reason for this? 微信图片_20240304111500

McGregorWwww commented 8 months ago

Hi, due to class-imbalance, the training of the Synapse dataset is highly unstable when using early-stopping (influenced by randomness), maybe you can change the validation interval to a smaller number, e.g., 5 or 2 or even 1 in https://github.com/xmed-lab/DHC/blob/975641335b902cd7fcd97ffc332ce97b0a6169ce/code/train_dhc.py#L364, or enlarge the early_stop_patience.

emma000730 commented 8 months ago

Is this data too low? The best evaluation dice is 0.13395972549915314. I modified the validation interval, but it seems it didn't help improve performance

McGregorWwww commented 8 months ago

Yes, it should be around 0.4, you can check our training logs in the weights downloading link.

McGregorWwww commented 8 months ago

Maybe you need to check whether the data is correctly processed, e.g., through visualization.

emma000730 commented 8 months ago

Thank you for your response. I have two more questions to ask: 1. The results of running another AMOS dataset are similar to yours, but the Dice score for the Synapse dataset is still around 0.15. Why is there such a difference when I move your dataset and code to my computer? Is this related to the small number of samples in the Synapse dataset? Would you give me some advice? 2. Why does the loss during training appear negative, even as large as -9 or -10?

McGregorWwww commented 8 months ago

  1. It is indeed due to the small number of samples in the Synapse dataset, there are several parameters you can modify: learning rate, batchsize, larger weight of unsupervised loss, smaller accumulate_iters of DiffDW and smaller momentum of DistDW.

  2. We apply -dice as the loss rather than 1-dice, so the loss can be negative; when weighted with the class weights, it can be smaller than -1.