Error occurs in XLMRobertaModel when token_type_ids is given.

dongjun-Lee commented 4 years ago

🐛 Bug

Model I am using (Bert, XLNet....): XLMRoberta

Language I am using the model on (English, Chinese....): English

The problem arise when using:

the official example scripts: (give details)

The tasks I am working on is:

[ ] an official GLUE/SQUaD task: (give the name)
[ ] my own task or dataset: (give details)

To Reproduce

Steps to reproduce the behavior:

>>> import torch
>>> from transformers import XLMRobertaModel
>>> model = XLMRobertaModel.from_pretrained('xlm-roberta-base', cache_dir="cache_dir")
>>> input_ids = torch.tensor([[0, 164, 100231, 135758, 32, 2, 2, 157, 217, 164, 10869, 5, 2]])
>>> outputs = model(input_ids)
>>> outputs[0].size()
torch.Size([1, 13, 768])
>>>
>>> token_type_ids = torch.tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]])
>>> outputs = model(input_ids, token_type_ids=token_type_ids)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 735, in forward
    embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 70, in forward
    inputs_embeds=inputs_embeds)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/transformers/modeling_bert.py", line 188, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/bering/anaconda3/envs/torch1.3/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 1 out of table with 0 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418

Expected behavior

Environment

OS: Ubuntu 18.04
Python version: 3.6
PyTorch version: 1.3
PyTorch Transformers version (or branch): 2.3.0
Using GPU ?: no
Distributed or parallel setup ?: no
Any other relevant information:

andompesta commented 4 years ago

XLMRobertaModel does not support token types > 0. If you look at the embedding you will see that there is only a single value. Basically the model does not rely on this embedding to understand when a sentence end. I think they included it only for API compatibility

dongjun-Lee commented 4 years ago

@andompesta Thank you very much! :)

huggingface / transformers