M3AE pre-trained model

[nooralahzadeh]

Hi, Could you share the pre-trained VLP models such as M3AE?

Thanks

[baeseongsu]

Hi @nooralahzadeh,

I've uploaded two pre-trained M3AE models used in our paper:

• m3ae_medicat_roco.ckpt: Reproduced version of the original M3AE model.
• m3ae_mimiccxr.ckpt: Pre-trained on a subset of MIMIC-CXR-JPG images. ~(This file will be uploaded soon, within 30 mins.)~

You can find them in this link (via Dropbox).

Best, Seongsu

[nooralahzadeh]

Hi @baeseongsu Thanks for the prompt answer and also for providing the checkpoints. It would be great if you could help me to find the grounTruth label set for the VQA task.

Thanks, Farhad

[baeseongsu]

Hi, @nooralahzadeh

You can download full VQA datasets with ground truth labels from here.

Best, Seongsu

[nooralahzadeh]

Thanks.

[nooralahzadeh]

Hello @baeseongsu , Thanks for providing the M3AE checkpoints. Could you please also provide the values you used to do inference using the m3ae_mimiccxr.ckp? Did you use this one as below for the VQA task:

python main.py with data_root=data/finetune_arrows/ \ num_gpus=${num_gpus} num_nodes=1 \ task_finetune_irtr_roco get_recall_metric=True \ per_gpu_batchsize=${per_gpu_batchsize} \ clip16 text_roberta \ image_size=288 \ test_only=True \ tokenizer=downloaded/roberta-base \ load_path=m3ae_mimiccxr.ckpt

[baeseongsu]

Hi @nooralahzadeh,

Yes, I've used a similar code script structure. However, the checkpoints I gave you are for the pre-trained VLP model, not fine-tuned on the MIMIC-CXR-VQA dataset. So I will upload several checkpoints which are fine-tuned models so that you can infer directly. You can check our experiment code for the M3AE model here.

[nooralahzadeh]

I would be grateful if you could upload the VQA checkpoints, we would like to reproduce your results.

[baeseongsu]

Hi @nooralahzadeh,

You can find them in this link (via Dropbox). There are in total 6 fine-tuned models on MIMIC-CXR-VQA with three seeds and two base pre-trained models. If you have any problems with the experiments, please feel free to reach out.

Best, Seongsu

[nooralahzadeh]

Thanks,