[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[x] My own task or dataset (give details below)
Reproduction
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained("openai-community/gpt2")
example = tokenizer("A test sentence", return_tensors="pt")
example = {k: v.flatten() for k, v in tensor_example.items()}
collator([example]*2)
Expected behavior
Collator should work with all output types supported by tokenizer.
System Info
latest transformers
Who can help?
@ArthurZucker
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
Collator should work with all output types supported by tokenizer.