Lower results obtained

[harpergith]

Hi, thank you for your efforts in organizing and releasing the code. I tried the code on the SIIM segmentation task by carefully following the instructions on organizing/splitting the dataset and setting up the open mmsegmentation framework. However, using the provided configuration files, the results cannot be reproduced.

The pre-trained weights MRM.pth are downloaded from the provided link. Finetuning on 100% SIIM data gave Dice 90.7% and finetuning on 10% data gave Dice 69.1%, which are lower than the values in the paper (91.4% and 73.2%) especially when finetuning on 10% data.

Could you clarify how to reproduce the results? Thanks.

[DopamineLcy]

Hi, thank you very much for following our work! I checked the training details of SIIM segmentation and adjusted some details of the learning rate and schedule, especially for finetuning on 10% data. I hope the adjustments are helpful, referring to https://github.com/RL4M/MRM-pytorch/tree/main/Siim_Segmentation/configs/mrm. Thank you again and welcome to ask new questions to help us improve the code.

[harpergith]

Thank you for your quick response. I tried to use the new configuration file "upernet_mrm-base_fp16_8x2_512x512_160k_siim_10per.py" and "schedule_5k" to finetune on 10% data, but got even lower results (67.03%) than before (69.1%).

[DopamineLcy]

I'm sorry for missing an important file when extracting core files from the complex code structure. See new commit(https://github.com/RL4M/MRM-pytorch/commit/694e67925f40e59af514819d439319142d2a1870) and (https://github.com/RL4M/MRM-pytorch/commit/96f7fc31749c1368a28b04b6a87cdefd2968067d) to get correct loss functions.

[funnyzhou]

@harpergith

Hi Harpergith,

Have you tried our new commit? Are there any problems with your experiments? If you can reproduce our results, we would close this issue now.

Best,

[harpergith]

Hi, Thank you for your quick efforts in improving the code. Now I can reproduce the results when finetuning on 10% data.

[harpergith]

Hi, as observed in the SIIM segmentation task, the finetuning configurations matter a lot on the performance. Do you have plan to release the fine-tuning code/configurations of other classification datasets, such as RSNA pneumonia?

[DopamineLcy]

Hi, the code and other hyperparameters are the same for other classification datasets. It may be helpful to see https://github.com/RL4M/MRM-pytorch/blob/3730ba49a678eda870284205178bfe25627aaed8/README.md?plain=1#L87 for the references of warm-up steps, total steps, and learning rate.

Thanks for your advice.

Best regards,

[harpergith]

Thank you for the update. This is very helpful.

Best regards,