Failed to reproduce "3D Self-Supervised Methods for Medical Imaging" results over the Pancreas dataset

[GuillaumeMougeot]

Dear @aihamtaleb,

I am currently trying to reproduce your baseline curve for the Pancreas dataset as well as the exemplar3D curve (Figure 3 of your NeurIPS publication). I am contacting you because I failed to find a script that allow me to get such curves on your Github page and, using the script that I wrote myself above the available code here, I failed to find a clear and significant difference between the baseline curve (trained in a fully supervised manner) and the exemplar3D curve (pretrained in a self-supervised manner and then retrained with supervision). I have not yet try to use the other methods.

I also preprocessed the Pancreas dataset myself, so it fits the expectations of your code (having, for instance, a fix size of 128x128x128) and I am not sure I did it in the right way.

I am thus contacting you today to ask you if you could help to reproduce this Figure 3 of your publication by sharing:

• the configuration script you used to trained your baseline model over the pancreas dataset?
• as well as the exact configuration script you used for the exemplar3D method? (the one presented in your Github repository seems incomplete as, for instance, the parameter "exp_splits" has the value [50,25] instead of [100,50,25,10])
• and finally, if possible, the preprocessed pancreas dataset?

Sorry for asking for so much, Many thanks in advance, Best, Guillaume Mougeot PhD Student

[aihamtaleb]

Dear @GuillaumeMougeot ,

Thanks for contacting us. Please let me know if you still have the issue.

All the configurations are mentioned explicitly in the paper (see the appendix too). Please try to carefully use these configs listed in the appendix.

Cheers, Aiham

[GuillaumeMougeot]

Dear @aihamtaleb,

Many thanks for your reply,

I tried to follow the paper and appendix indications and after many tries, I managed to reproduce your results with the RPL 3D method over the pancreas dataset that I pre-processed myself. However, when runnning exactly the same procedure with the examplar 3D method I did not manage to reproduce your results (I don't see a significant difference between a supervised and a self-supervised training). As I don't really know how to explain my problem more than that, I am not sure it is relevant to keep this issue open so I will close it... Thank you again for your time.

Cheers, Guillaume