question about the initialization of teacher model

[dazhangyu123]

I am reimplementing your method on a new dataset and want to know how to get the initialization checkpoint of the teacher model.

[DearCaat]

just regular training but the random seed should be matched, e.g., baseline: transmil, seed: 2021: python3 main.py --project=$PROJECT_NAME --dataset_root=$DATASET_PATH --model_path=$OUTPUT_PATH --cv_fold=3 --title=transmil --model=transmil --seed=2021

[dazhangyu123]

I tried your suggestion, but I encountered the issue of "Missing keys & unexpected keys" mismatch when loading the teacher_init checkpoint. To work around this problem, I executed the following code: python3 main.py --project=$PROJECT_NAME --dataset_root=$DATASET_PATH --model_path=$OUTPUT_PATH --cv_fold=3 --title=transmil --model=pure --baseline=selfattn --seed=2021 I am uncertain whether this operation is correct. Could you please confirm if this approach is appropriate, or if there is a better solution to address the issue? Thank you.

[DearCaat]

Oh, that's really the right way to do it. Sorry I got it wrong before. Actually, the pure model is essentially the baseline model, except that it is compatible with the student model. It is worth noting that we set the n_heads of transmil on the tcga dataset to 2.

[akidway]

Hi, @DearCaat Can I directly use baseline dsmil weight as teacher_init for dsmil mhim (same random seed)? From your code, I found only one difference between baseline dsmil and dsmil for mhim is dsmil for mhim in i_classifier miss a nn.drop().