search

liupei101 / AdvMIL

[MedIA 2024] The implementation of AdvMIL: Adversarial Multiple Instance Learning for the Survival Analysis on Whole-Slide Images
MIT License
29 stars 10 forks source link

High c-index for patchGCN on LUAD dataset #2

Closed Raymvp closed 1 year ago

Raymvp commented 1 year ago

Dear Dr. Liu, I came across a unique situation while working on a replication of patchGCN, which I noticed you utilized in the experimental section of your recent paper.

In the process of my experiments, I observed a relatively high c-index for the validation set on the LUAD dataset, around 0.75, while the c-index during the training phase was around 0.68. This seemed unusual as the original paper reported a c-index of only 0.585.

Did you encounter any anomalies or unusually high c-index values in your implementations on patchGCN? Any insights you could provide would be immensely helpful and appreciated.

Thank you for your time

Xin Liu

liupei101 commented 1 year ago

Our experimental results are roughly the same as those reported in the paper of PatchGCN. If there is a significant gap between your results and the original ones, you could check the following points, in my opinion,

  1. make sure that you have followed the same experimental setting as PatchGCN, e.g., using 5-fold cross-validation, etc.
  2. from my observations, a large C-Index could be obtained if you set a relatively-small time bin (that is used to uniformly divide survival time into several bins).

Hope these could help you. Good luck!

Raymvp commented 1 year ago

The default time bin is 4, but I've tried setting it to 8 and 10, and the results didn't show significant differences. Could you please tell me what's the time bin you've set? If it's possible, could you share your patchGCN code with me? Thank you.

liupei101 commented 1 year ago

Our setting is also 4. Our codes of PatchGCN are available at here.

It is highly recommended that you may ask for help at the official site of PatchGCN. You would get more useful suggestions and insights from the authors of PatchGCN for the questions you raised here.

Good luck!

Raymvp commented 1 year ago

刘博士你好！首先感谢您之前的帮助！我已经在PatchGCN页面的issue问过了，没有得到有效的回答。于是尝试用您的代码来跑。 现在我想知道如何能成功用您的代码跑通patchGCN。有几个缺失的参数，我设置为pdh_dims: 384-1 mlp_hops: 3 mlp_norm: True mlp_dropout: 0.25 opt_net: adam opt_net_weight_decay: 1e-4 opt_net_lr: 0.0002，但是并不是很有效，我用4090显卡跑，batchsize为1的情况下，报错RuntimeError: CUDA out of memory. Tried to allocate 1.11 GiB (GPU 5; 23.65 GiB total capacity; 21.38 GiB already allocated; 964.56 MiB free; 21.61 GiB reserved in total by PyTorc h) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 。 我想请问,patchGCN的yaml应该如何设置？

liupei101 commented 1 year ago

我使用patchgcn也出现过类似问题。据之前的实验经历，建议您检查这样几个实验参数：

  1. 降低每个batch 内patch的总数，一般1500左右是没问题的。
  2. 设置GCN层数为1。
  3. 降低embedding 的维度，比如把384改为128，或256。
Raymvp commented 1 year ago

我在GCN层数和原始论文一样，bp_every_batch: 3的情况下，用您的代码依然跑出了超出原文与您论文复现中的效果。 image image 是否可能是由于 代码依赖包 的更新，导致的效果大幅上升？

liupei101 commented 1 year ago

数据的准备，比如patch划分的设置，以及patch特征提取的设置等，也可能会影响最终的预测性能。您能提供一下上面运行结果相关的完整的yaml文件配置吗？

Raymvp commented 1 year ago

好的。task: surv_nll # cont_gansurv seed: 42 cuda_id: 5

wandb_dir: /home/ubuntu/AdvMIL-main # path to this repo wandb_prj: patchGCN_defult # wandb project name save_path: ./results-adv/brca-patchGCN # path to save log files during training and testing

data

dataset: BRCA path_patch: /home/ubuntu/dataset/TCGA_BRCA_features/extracted_feature/pt_files # path to patch features, for patch-based models path_graph: /home/ubuntu/dataset/TCGA_BRCA_features/tcga_brca_20x_features/graph_euclidean_files # path to WSI graphs, for graph-based models path_cluster: /data/nlst/processed/patch-l1-cluster8-ids # path to patch clusters, for cluster-based models path_coordx5: null path_label: ./table/tcga_brca_path_full.csv # path to the csv table with patient_id, pathology_id, t, e feat_format: pt time_format: ratio time_bins: 4 data_split_path: ./data_split/tcga_brca-fold{}.npz # path to data split data_split_seed: [0, 1, 2, 3, 4] # fold identifiers, used to fill the placeholder in data_split_path save_prediction: True train_sampling: null # w/o data sampling

Backbone setting of MIL encoder

bcb_mode: graph # choose patch, cluster, or graph bcb_dims: 1024-128-128 # the dims from input dim -> hidden dim -> embedding dim

以下部分自己设置

pdh_dims: 128-4 mlp_hops: 1 mlp_norm: True mlp_dropout: 0.25 opt_net: adam opt_net_weight_decay: 0.00001 opt_net_lr: 0.0002

cuda_id: 5

Generator setting (regarding the end part)

gen_dims: 128-1 # embedding dim -> out dim gen_noi_noise: 0-1 # noise setting, 0-1 / 1-0 / 1-1 gen_noi_noise_dist: uniform # noise type, gaussian / uniform gen_noi_hops: 1 gen_norm: False gen_dropout: 0.6 gen_out_scale: sigmoid

Discriminator

disc_type: prj # how to fuse X and t: cat (vector concatenation) / prj (vector projection) disc_netx_in_dim: 1024 # input dim of X disc_netx_out_dim: 128 # out dim of X disc_netx_ksize: 1 disc_netx_backbone: avgpool disc_netx_dropout: 0.25 disc_nety_in_dim: 1 # input dim of t disc_nety_hid_dims: 64-128 # hidden dim of t disc_nety_norm: False disc_nety_dropout: 0.0 disc_prj_path: x disc_prj_iprd: bag # choose bag (regular projection) / instance (RLIP)

loss for all

loss_gan_coef: 0.004 # coefficient of GANLoss loss_netD: bce # choose bce / hinge / wasserstein loss_regl1_coef: 0.00001 # coefficient of L1 Regularization

loss for discrete model

loss_mle_alpha: 0.0

loss for continuous model

loss_recon_norm: l1 # l1/l2 loss_recon_alpha: 0.0 loss_recon_gamma: 0.0

Optimizer

opt_netG: adam opt_netG_lr: 0.00008 # learning rate of generator opt_netG_weight_decay: 0.0005 opt_netD_lr: 0.00008 # learning rate of discriminator

training

epochs: 20 # epoch numbers batch_size: 1 bp_every_batch: 3 num_workers: 8 # work numbers for loading WSI features es_patience: 3 # es: early stopping es_warmup: 5 es_verbose: True es_start_epoch: 0 gen_updates: 1 # 1/2 monitor_metrics: loss # metrics on validation set for early stopping

test

times_test_sample: 30 # sampling times when predicting survival from each WSI. log_plot: False

Raymvp commented 1 year ago

这是我的环境所用的包 name: clam channels:

Raymvp commented 1 year ago

我使用patchgcn也出现过类似问题。据之前的实验经历，建议您检查这样几个实验参数：

  1. 降低每个batch 内patch的总数，一般1500左右是没问题的。
  2. 设置GCN层数为1。
  3. 降低embedding 的维度，比如把384改为128，或256。

刘博士您好，我最近又在做这方面的工作。我还是发现默认设置下，patchGCN容易爆显存，我用的是一张4090，值得一提的是，您的实验是使用2xV100，原作者是使用4x2080，我有个疑问，在batchsize=1单张显卡爆显存的情况下，是否有办法叠加多张显卡，解决爆显存的问题。

sjjadsa commented 3 months ago

@Raymvp 你好。请问你用LUAD数据集时，data_spilt是怎么划分的？