szc19990412
commented
2 years ago - For the magnification problems in CAMELYON16, we just uesd CLAM like
python create_patches_fp.py --source DATA_DIRECTORY --save_dir RESULTS_DIRECTORY --patch_level 1 --patch_size 256 --seg --patch --stitch. In my view, 40 times of pathological image corresponds to specimen-level pixel size 0.25 μm, and 20 times corresponds to specimen-level pixel size 0.5 μm. So maybe we need to use patches all in 0.5 μm. - CAMELYON16 has an official test dataset, so we make train-val split in training dataset to select best model, and report the best fold result. It's OK you can also test how the model performs under your experimental setup.
- For the ABMIL, because all the Compared model used the patches feature. So our implemention was based on Attn_Net in CLAM.
- EarlyStopping with patience=10 on val_loss, that's OK. More patience sometimes will be better, like 30.
Tonyboy999
Hi all,
Thanks for your work. All looks good except I have been trying for some time to reproduce your results. I just discovered the thread https://github.com/szc19990412/TransMIL/issues/4 where others also have failed to reproduce the results. More specifically, I have been trying to reproduce the CAMELYON16 TransMIL ablations and the ABMIL benchmark using your repository and my own repository (written using PyTorch Lightning).
Some things I would like to know for reproducibility of the experiments:
The given CAMELYON16 fold is only one fold, based on a 5:1 train-val split. In your paper you report using 10:1 train-val splits. I would therefore like to know what splits you used and whether they were stratified or not. For reproducibility I would like to request all splits to be uploaded to the repository so people are working with the same data.
The CAMELYON16 dataset contains slides that were sourced from two centers that used different magnifications for each scanner (https://jamanetwork.com/journals/jama/fullarticle/2665774). If 20x magnification was used for the whole dataset, could you please confirm whether the slides from these two centers were tiled at two different physical resolutions (microns per pixel)?
“The whole-slide images were acquired at 2 different centers using 2 different scanners. RUMC images were produced with a digital slide scanner (Pannoramic 250 Flash II; 3DHISTECH) with a 20x objective lens (specimen-level pixel size, 0.243 μm × 0.243 μm). UMCU images were produced using a digital slide scanner (NanoZoomer-XR Digital slide scanner C12000-01; Hamamatsu Photonics) with a 40x objective lens (specimen-level pixel size, 0.226 μm × 0.226 μm).”The model selection/checkpointing details. I see from the code https://github.com/szc19990412/TransMIL/blob/3f6bbe868ac39e7d861a111398b848ba3b943ca8/utils/utils.py#L43-L49, that you used
EarlyStoppingwithpatience=10onval_loss, correct?What optimizer and hyperparameters were used for ABMIL? I assume what the authors (https://arxiv.org/pdf/1802.04712.pdf) report in Table 17 for the histopathology datasets?
I will later report my scores for TransMIL & ABMIL using the repository and given data fold. Thanks in advance.