TypeError: '<' not supported between instances of 'NoneType' and 'int'

[ruben-soria-herrero]

https://github.com/huggingface/transformers/blob/96429e74a8191521bcb4b99f48ad1fbc8f9e6873/src/transformers/tokenization_utils_base.py#L2921

It returns this error when launching tgi with Meta-Llama-3.1-8B-Instruct:

ERROR text_generation_launcher: Method Warmup encountered an error. Traceback (most recent call last): File "/opt/conda/bin/text-generation-server", line 8, in sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in call return get_command(self)(*args, kwargs) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(*use_params) # type: ignore File "/opt/conda/lib/python3.10/site-packages/text_generation_server/cli.py", line 109, in serve server.serve( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 274, in serve asyncio.run( File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete self.run_forever() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args) File "/opt/conda/lib/python3.10/site-packages/grpc_interceptor/server.py", line 165, in invoke_intercept_method return await self.intercept(

File "/opt/conda/lib/python3.10/site-packages/text_generation_server/interceptor.py", line 21, in intercept return await response File "/opt/conda/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 120, in _unary_interceptor raise error File "/opt/conda/lib/python3.10/site-packages/opentelemetry/instrumentation/grpc/_aio_server.py", line 111, in _unary_interceptor return await behavior(request_or_iterator, context) File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 120, in Warmup batch = self.model.batch_type.from_pb( File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/causal_lm.py", line 119, in from_pb tokenized_inputs = tokenizer( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3073, in call encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3160, in _call_one return self.batch_encode_plus( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3347, in batch_encode_plus padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( File "/opt/conda/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2976, in _get_padding_truncation_strategies if padding_strategy != PaddingStrategy.DO_NOT_PAD and (self.pad_token is None or self.pad_token_id < 0): TypeError: '<' not supported between instances of 'NoneType' and 'int' ERROR warmup{max_input_length=4095 max_prefill_tokens=4145 max_total_tokens=4096 max_batch_size=None}:warmup: text_generation_router_v3::client: backends/v3/src/client/mod.rs:54: Server error: '<' not supported between instances of 'NoneType' and 'int' Error: Backend(Warmup(Generation("'<' not supported between instances of 'NoneType' and 'int'")))

self.pad_token is defined as str or tokenizers.AddedToken, so I think it can be not None (but be a NoneType of an AddedToken) and throw a TypeError when compared with an 'int'.

[LysandreJik]

cc @itazap :)

[itazap]

hey @ruben-soria-herrero ! Can you please share a small reproducer to replicate? 😊

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.