search

taokz / BiomedGPT

BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks
Apache License 2.0
363 stars 34 forks source link

Peir Gross Image Captioning - /type2ans.json' not found and in zero shot it throwing unsupported Error #23

Closed sreelakshminarayanan closed 3 months ago

sreelakshminarayanan commented 3 months ago

Hi Team,

We are trying to evaulate image captioning for peir gross, we are getting the below error. Similar issue has been reported earlier, zer-shot has been suggested, for image captioning it is throwing unsuported error.

https://github.com/taokz/BiomedGPT/issues/19

File "../../evaluate.py", line 162, in cli_main() File "../../evaluate.py", line 157, in cli_main cfg, main, ema_eval=args.ema_eval, beam_search_vqa_eval=args.beam_search_vqa_eval, zero_shot=args.zero_shot File "/home/ubuntu/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/fairseq/distributed/utils.py", line 354, in call_main distributed_main(cfg.distributed_training.device_id, main, cfg, kwargs) File "/home/ubuntu/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/fairseq/distributed/utils.py", line 328, in distributed_main main(cfg, kwargs) File "../../evaluate.py", line 82, in main num_shards=cfg.checkpoint.checkpoint_shard_count, File "/home/ubuntu/BiomedGPT/utils/checkpoint_utils.py", line 447, in load_model_ensemble_and_task task = tasks.setup_task(cfg.task) File "/home/ubuntu/miniconda3/envs/biomedgpt/lib/python3.7/site-packages/fairseq/tasks/init.py", line 46, in setup_task return task.setup_task(cfg, kwargs) File "/home/ubuntu/BiomedGPT/tasks/ofa_task.py", line 111, in setup_task return cls(cfg, src_dict, tgt_dict) File "/home/ubuntu/BiomedGPT/tasks/pretrain_tasks/unify_task.py", line 97, in init self.type2ans_dict = json.load(open(os.path.join(self.cfg.neg_sample_dir, 'type2ans.json'))) FileNotFoundError: [Errno 2] No such file or directory: '/data/omnimed/pretrain_data/negative_sample/type2ans.json'

Below is the script screenshot

image

sreelakshminarayanan commented 3 months ago

Could you please assit on this issue

taokz commented 3 months ago

You can convert the test.tsv from captioning format to VQA format, or modify the selected_cols in the script as well as set task=vqa.

sreelakshminarayanan commented 3 months ago

Thanks @taokz for the response. But for our study, we are planning to implement image captioning for IU Xray or perigross, which may not be able to leverage vqa. I checked the vqa sample data format, it seems to have a question which is applicable for VQA RAD, it is not available for peir gross.

taokz commented 3 months ago

Image captioning has a system prompt with "What does the image describe?" as described in the paper, which could be taken as the question to build a VQA format. Besides, you can also make the prompt more diverse with the help of chatgpt or manually design.

sreelakshminarayanan commented 3 months ago

Thanks @taokz for the quick response , will build it