search

Event-AHU / Medical_Image_Analysis

Foundation models based medical image analysis
51 stars 0 forks source link

iu_xray提供的checkpoint无法跑通demo #3

Open hitsz-zxw opened 1 month ago

hitsz-zxw commented 1 month ago

你好，很感谢你们做出的贡献，我对你们的工作很感兴趣，也跑通了你们提供的demo代码，但是，当我把模型从提供的mimic.pth换成iu.pth时，出现了以下错误： Loading vision encoder:/disk0/uni_group/Medical_Image_Analysis/R2GenCSR/VMamba/vssm_base_0229_ckpt_epoch_237.pth => merge config from ./VMamba/classification/configs/vssm1/vssm_base_224.yaml vssm => loaded pretrinaed vssm model successfully Loading Trainable vision encoder:/disk0/uni_group/Medical_Image_Analysis/R2GenCSR/VMamba/vssm_base_0229_ckpt_epoch_237.pth -- Done Loading LLAMA llama2 Loading checkpoint shards: 100%|███████████████████████████████████████████████████████| 2/2 [00:01<00:00, 1.31it/s] Loading LLAMA Done Traceback (most recent call last): File "/disk0/uni_group/Medical_Image_Analysis/R2GenCSR/demo.py", line 14, in model = R2GenCSR(args) File "/disk0/uni_group/ck/xiwen/Medical_Image_Analysis/R2GenCSR/models/R2GenCSR_domo.py", line 190, in init self.load_state_dict(state_dict=state_dict, strict=False) File "/disk0/uni_group/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2153, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for R2GenCSR: size mismatch for llama_proj.weight: copying a param with shape torch.Size([2048, 1024]) from checkpoint, the shape in current model is torch.Size([4096, 1024]). size mismatch for llama_proj.bias: copying a param with shape torch.Size([2048]) from checkpoint, the shape in current model is torch.Size([4096]). size mismatch for layer_norm.weight: copying a param with shape torch.Size([2048]) from checkpoint, the shape in current model is torch.Size([4096]). size mismatch for layer_norm.bias: copying a param with shape torch.Size([2048]) from checkpoint, the shape in current model is torch.Size([4096]).

liyhc commented 1 month ago

这是因为iu.pth用的是iu-xray数据集训练配置，语言模型是Qwen不是Llama，另外各项参数得和iu.sh配置一样。

hitsz-zxw commented 1 month ago

十分感谢你的回答，现在我已经能够跑通iu_xray，还有一个问题，当我运行测试脚本iu.sh时，出现了以下报错

Traceback (most recent call last): File "/disk0/Medical_Image_Analysis/R2GenCSR/train.py", line 55, in main() File "/disk0/Medical_Image_Analysis/R2GenCSR/train.py", line 51, in main train(args) File "/disk0/Medical_Image_Analysis/R2GenCSR/train.py", line 40, in train trainer.test(model, datamodule=dm) File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 735, in test return call._call_and_handle_interrupt( File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 41, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, kwargs) File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/strategies/launchers/subprocess_script.py", line 91, in launch return function(*args, *kwargs) File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 778, in _test_impl results = self._run(model, ckpt_path=ckpt_path) File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 973, in _run results = self._run_stage() File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 1009, in _run_stage return self._evaluation_loop.run() File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/loops/utilities.py", line 177, in _decorator return loop_run(self, args, kwargs) File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 122, in run return self.on_run_end() File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 244, in on_run_end self._on_evaluation_epoch_end() File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/loops/evaluation_loop.py", line 326, in _on_evaluation_epoch_end call._call_lightning_module_hook(trainer, hook_name) File "/disk0/miniconda3/envs/R2GenCSR/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 144, in _call_lightning_module_hook output = fn(*args, **kwargs) File "/disk0/Medical_Image_Analysis/R2GenCSR/models/R2GenCSR.py", line 795, in on_test_epoch_end eval_res = self.score(ref=ref,hypo=hypo) File "/disk0/Medical_Image_Analysis/R2GenCSR/models/R2GenCSR.py", line 219, in score score, scores = scorer.compute_score(ref, hypo) File "/disk0/Medical_Image_Analysis/R2GenCSR/evalcap/meteor/meteor.py", line 77, in compute_score stat = self._stat(res[i][0], gts[i]) File "/disk0/Medical_Image_Analysis/R2GenCSR/evalcap/meteor/meteor.py", line 108, in _stat self.meteor_p.stdin.flush() BrokenPipeError: [Errno 32] Broken pipe

然后我又尝试了运行你们提供的test_meteor.py，也是同样的报错BrokenPipeError，这似乎是Java评测脚本的问题，我该如何解决？或者我有什么方法可以得到模型生成的结果吗，这样我可以不通过Java库通过python的比如pycocoevalcap库来评测，以这样的方法能够复现你们论文中提到的结果吗？或者我通过逐个sample调用函数demo_test_step能得到相同的结果吗？

liyhc commented 1 month ago

得下载 paraphrase-en.gz here , 将文件放在meteor/data文件夹中