Unable to reproduce Camelyon16 results

[Yew-yy]

@szc19990412 Thank you very much for your excellent work, it has given me a lot of inspiration. However I had some problems reproducing the results mentioned in your paper. I browsed through the previous issues and I trained the camelyon16 data exactly the way you mentioned, including using a 20x patch, setting patience=10 or 30, and dividing the training and validation sets in a 10:1 ratio. Nevertheless, on the test set I obtained similar results to those mentioned in issue 4. The results are far from those mentioned in the paper. Is there any training detail that I missed, if so, please let me know, thanks a lot. test_Accuracy=0.7109375 test_CohenKappa=0.37618548, test_Recall=0.6844485 test_Precision=0.6931818 auc=0.6600362

[yyyyxxxyyyy]

@szc19990412 Thank you very much for your excellent work, it has given me a lot of inspiration. However I had some problems reproducing the results mentioned in your paper. I browsed through the previous issues and I trained the camelyon16 data exactly the way you mentioned, including using a 20x patch, setting patience=10 or 30, and dividing the training and validation sets in a 10:1 ratio. Nevertheless, on the test set I obtained similar results to those mentioned in issue 4. The results are far from those mentioned in the paper. Is there any training detail that I missed, if so, please let me know, thanks a lot. test_Accuracy=0.7109375 test_CohenKappa=0.37618548, test_Recall=0.6844485 test_Precision=0.6931818 auc=0.6600362 Dear ：

I hope this message finds you well. I am writing to you because I am a beginner in MIL and I am having some difficulty understanding it. I came across your review and I was wondering if you could provide me with some guidance on how to reproduce this code.

Would you mind sharing with me the detailed steps that you took to reproduce the code? I am particularly interested in understanding the process from start to finish, including any necessary pre-processing steps, data selection, and any other important details that may be relevant.

I appreciate any assistance you can provide and I look forward to hearing back from you soon.

Best regards

[lingxitong]

@szc19990412 Thank you very much for your excellent work, it has given me a lot of inspiration. However I had some problems reproducing the results mentioned in your paper. I browsed through the previous issues and I trained the camelyon16 data exactly the way you mentioned, including using a 20x patch, setting patience=10 or 30, and dividing the training and validation sets in a 10:1 ratio. Nevertheless, on the test set I obtained similar results to those mentioned in issue 4. The results are far from those mentioned in the paper. Is there any training detail that I missed, if so, please let me know, thanks a lot. test_Accuracy=0.7109375 test_CohenKappa=0.37618548, test_Recall=0.6844485 test_Precision=0.6931818 auc=0.6600362

maybe some tricks had not shown in the paper

[weiaicunzai]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

[lingxitong]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

[weiaicunzai]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

Thanks, glad to hear. Did you set shuffle in the class CamelData to True during training? I didn't noticed that until now, the default shuffle argument is False data_shuffle: False in the file TransMIL.yaml.

[lingxitong]

i use the origin setting 

---Original--- From: @.> Date: Sat, Jan 6, 2024 16:59 PM To: @.>; Cc: @.**@.>; Subject: Re: [szc19990412/TransMIL] Unable to reproduce Camelyon16 results(Issue #25)

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

Thanks, glad to hear. Did you set shuffle in the class CamelData to True during training? I didn't noticed that until now, the default shuffle argument is False data_shuffle: False in the file TransMIL.yaml.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[weiaicunzai]

i use the origin setting  … ---Original--- From: @.> Date: Sat, Jan 6, 2024 16:59 PM To: @.>; Cc: @.**@.>; Subject: Re: [szc19990412/TransMIL] Unable to reproduce Camelyon16 results(Issue #25) I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper. I can get 0.87,should use early stop Thanks, glad to hear. Did you set shuffle in the class CamelData to True during training? I didn't noticed that until now, the default shuffle argument is False data_shuffle: False in the file TransMIL.yaml. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Thanks, could you please tell me how do you process the Camelyon16 dataset? I think the performance gap between our results might be caused by the different way of processing the training data. The default settings of CLAM api doesn't perform well in the Camelyon16 dataset . some slides can not even generate patches, So I use the training data released by DTFD-MIL to train trans-mil.

[lingxitong]

lingxitong vchat

---Original--- From: @.> Date: Sat, Jan 6, 2024 17:19 PM To: @.>; Cc: @.**@.>; Subject: Re: [szc19990412/TransMIL] Unable to reproduce Camelyon16 results(Issue #25)

i use the origin setting  … ---Original--- From: @.> Date: Sat, Jan 6, 2024 16:59 PM To: @.>; Cc: @.@.>; Subject: Re: [szc19990412/TransMIL] Unable to reproduce Camelyon16 results(Issue #25) I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper. I can get 0.87,should use early stop Thanks, glad to hear. Did you set shuffle in the class CamelData to True during training? I didn't noticed that until now, the default shuffle argument is False data_shuffle: False in the file TransMIL.yaml. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Thanks, could you please tell me how do you process the Camelyon16 dataset? I think the performance gap between our results might be caused by the different way of processing the training data. The default settings of CLAM api doesn't perform well in the Camelyon16 dataset . some slides can not even generate patches, So I use the training data released by DTFD-MIL to train trans-mil.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[weiaicunzai]

I can get 0.87,should use early stop

lingxitong vchat … ---Original--- From: @.> Date: Sat, Jan 6, 2024 17:19 PM To: @.>; Cc: @.**@.>; Subject: Re: [szc19990412/TransMIL] Unable to reproduce Camelyon16 results(Issue #25) i use the origin setting  … ---Original--- From: @.> Date: Sat, Jan 6, 2024 16:59 PM To: @.>; Cc: @.@.>; Subject: Re: [szc19990412/TransMIL] Unable to reproduce Camelyon16 results(Issue #25) I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper. I can get 0.87,should use early stop Thanks, glad to hear. Did you set shuffle in the class CamelData to True during training? I didn't noticed that until now, the default shuffle argument is False data_shuffle: False in the file TransMIL.yaml. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.> Thanks, could you please tell me how do you process the Camelyon16 dataset? I think the performance gap between our results might be caused by the different way of processing the training data. The default settings of CLAM api doesn't perform well in the Camelyon16 dataset . some slides can not even generate patches, So I use the training data released by DTFD-MIL to train trans-mil. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

I'm sorry, but what do you mean by 'vchat' ? You manually labeled the tissue areas yourself?

[weiaicunzai]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

After setting shuffle to True, I'm getting acc around 0.84. I believe 0.87 is very feasible.

[lingxitong]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

After setting shuffle to True, I'm getting acc around 0.84. I believe 0.87 is very feasible.

wechat:lingxitong

[weiaicunzai]

vchat

i use the origin setting 

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

After setting shuffle to True, I'm getting acc around 0.84. I believe 0.87 is very feasible.

wechat:lingxitong

Thanks, I have sent the friend request.

[homadnmr]

@szc19990412 Thank you very much for your excellent work, it has given me a lot of inspiration. However I had some problems reproducing the results mentioned in your paper. I browsed through the previous issues and I trained the camelyon16 data exactly the way you mentioned, including using a 20x patch, setting patience=10 or 30, and dividing the training and validation sets in a 10:1 ratio. Nevertheless, on the test set I obtained similar results to those mentioned in issue 4. The results are far from those mentioned in the paper. Is there any training detail that I missed, if so, please let me know, thanks a lot. test_Accuracy=0.7109375 test_CohenKappa=0.37618548, test_Recall=0.6844485 test_Precision=0.6931818 auc=0.6600362

HI as I can see you could run the code. May I ask you to share the exact environment package versions?

[homadnmr]

vchat

i use the origin setting

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

After setting shuffle to True, I'm getting acc around 0.84. I believe 0.87 is very feasible.

wechat:lingxitong

Thanks, I have sent the friend request.

HI as I can see you could run the code. May I ask you to share the exact environment package versions?

[homadnmr]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

I can get 0.87,should use early stop

After setting shuffle to True, I'm getting acc around 0.84. I believe 0.87 is very feasible.

wechat:lingxitong

Hi :) as I can see you could run the code. May I ask you to share the exact environment package versions?

[homadnmr]

HI as I can see you could run the code. May I ask you to share the exact environment package versions?

[homadnmr]

I can get around 0.78 - 0.79 acc, however, still 10 percent less than reported in the paper.

Hi as I can see you could run the code. May I ask you to share the exact environment package versions?

[homadnmr]

@szc19990412 Thank you very much for your excellent work, it has given me a lot of inspiration. However I had some problems reproducing the results mentioned in your paper. I browsed through the previous issues and I trained the camelyon16 data exactly the way you mentioned, including using a 20x patch, setting patience=10 or 30, and dividing the training and validation sets in a 10:1 ratio. Nevertheless, on the test set I obtained similar results to those mentioned in issue 4. The results are far from those mentioned in the paper. Is there any training detail that I missed, if so, please let me know, thanks a lot. test_Accuracy=0.7109375 test_CohenKappa=0.37618548, test_Recall=0.6844485 test_Precision=0.6931818 auc=0.6600362

Hi :) as I can see you could run the code. May I ask you to share the exact environment package versions?