ValueError: Input <|endoftext|> is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.

[knhaller]

When running demo.py with 'gpt2' as the model, I came across this issue:

Loading ModelTraceback (most recent call last): File "demo.py", line 620, in break_tokens = tokenizer.encode(tokenizer._eos_token) + tokenizer.encode('?') + tokenizer.encode('!') File "/balboa/projects/conv_ai/packages/transformers/src/transformers/tokenization_utils_base.py", line 1430, in encode kwargs, File "/balboa/projects/conv_ai/packages/transformers/src/transformers/tokenization_utils_base.py", line 1742, in encode_plus kwargs, File "/balboa/projects/conv_ai/packages/transformers/src/transformers/tokenization_utils.py", line 454, in _encode_plus first_ids = get_input_ids(text) File "/balboa/projects/conv_ai/packages/transformers/src/transformers/tokenization_utils.py", line 442, in get_input_ids f"Input {text} is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers." tmp = b ValueError: Input <|endoftext|> is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.

The error was fixed when I changed line 620 to: break_tokens = tokenizer.encode(tokenizer.eos_token) + tokenizer.encode('?') + tokenizer.encode('!')

[fasterbuild]

tokenizer._eos_token->tokenizer.eos_token