search

ge-xing / SegMamba

SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation
https://arxiv.org/abs/2401.13560
332 stars 17 forks source link

`get_train_val_test_loader_from_train` does not produce consistent splits across machines #43

Open crrrr30 opened 2 months ago

crrrr30 commented 2 months ago

First off, thanks for sharing your complete code, which has been more than helpful for my current research.

Observation: It was rather challenging to have identified this issue. In particular, I noted that evaluating on a different machine than training produces unrealistically high Dice and HD95 metrics. Evaluation results, however, remain reproducible on the same machine.

Problem: According to this post, the return of glob.glob is arbitrary. Indeed, I could confirm that the return of glob.glob in line 247 is different on different machine, although on the same machine it seems to consistently reproduce the same ordering.

Could you share your data split in the paper so that we can reproduce your results? Thanks in advance for your time and attention.

crrrr30 commented 2 months ago

I'll add that it seems to be dependent on the file system. If the machine access a shared directory containing the data, then the training split is reproducible. Otherwise, if different machines have different copies of data (i.e., potentially different metadata), they do not in general output the same glob.glob() results.