search

tjvsonsbeek / open-ended-medical-vqa

Repository for the paper: Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models (https://arxiv.org/abs/2303.05977)
14 stars 1 forks source link

Tokens and masks #1

Closed JuanLyons closed 11 months ago

JuanLyons commented 11 months ago

Hi i'm trying to run the code. The command is a normal one. Python main.py.

Traceback (most recent call last):
  File "/media/disk2/jclyons/AML/Project/open-ended-medical-vqa/main.py", line 86, in <module>
    model = pytorch_model_run(train_dataloader, val_dataloader, model, args)
  File "/media/disk2/jclyons/AML/Project/open-ended-medical-vqa/train.py", line 71, in pytorch_model_run
    outputs = model(prefix, labels, tokens, mask, q_len, batch_size=args.batch_size)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/media/disk2/jclyons/AML/Project/open-ended-medical-vqa/models.py", line 31, in forward
    return self.gpt(inputs_embeds=embedding, attention_mask=mask)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1076, in forward
    transformer_outputs = self.transformer(
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 900, in forward
    outputs = block(
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 390, in forward
    attn_outputs = self.attn(
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 331, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/jclyons/anaconda3/envs/vqa/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 206, in _attn
    attn_weights = attn_weights + attention_mask
RuntimeError: The size of tensor a (46) must match the size of tensor b (50301) at non-singleton dimension 3

I think it could be an error in the dataloader.py file because of the make padding function.

JuanLyons commented 11 months ago

In the dataloader.py file you can chage this part

m_mask = [torch.ones(len(self.tokenizer.encode('question: '))),torch.ones(len(self.tokenizer.encode(' context:'))),torch.ones(len(self.tokenizer.encode('answer '))),torch.zeros(len(self.tokenizer.encode('<|endoftext|>')))]
YFan07 commented 9 months ago

In the dataloader.py file you can chage this part

m_mask = [torch.ones(len(self.tokenizer.encode('question: '))),torch.ones(len(self.tokenizer.encode(' context:'))),torch.ones(len(self.tokenizer.encode('answer '))),torch.zeros(len(self.tokenizer.encode('<|endoftext|>')))]

Hi, I have the same problem. Do you know how to modify it specifically? Hope to get your reply, thank you!!!