search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.15k stars 897 forks source link

Whisper example crashes with English-only models #1960

Open mrdrprofuroboros opened 1 month ago

mrdrprofuroboros commented 1 month ago

System Info

just a simple python bug, system agnostic

Who can help?

@byshiue

Information

Tasks

Reproduction

follow the official whisper tensorrt guide, but use small.en model

Expected behavior

inference work

actual behavior

Traceback (most recent call last): File "/home/jovyan/whisper/run.py", line 363, in model = WhisperTRTLLM(args.engine_dir, args.debug, args.assets_dir) File "/home/jovyan/whisper/run.py", line 236, in init tokenizer_name == "gpt2" UnboundLocalError: local variable 'tokenizer_name' referenced before assignment

additional notes

Here's the fix https://github.com/NVIDIA/TensorRT-LLM/pull/1958

yuekaizhang commented 1 month ago

System Info

just a simple python bug, system agnostic

Who can help?

@byshiue

Information

  • [x] The official example scripts
  • [ ] My own modified scripts

Tasks

  • [x] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • [ ] My own task or dataset (give details below)

Reproduction

follow the official whisper tensorrt guide, but use small.en model

Expected behavior

inference work

actual behavior

Traceback (most recent call last): File "/home/jovyan/whisper/run.py", line 363, in model = WhisperTRTLLM(args.engine_dir, args.debug, args.assets_dir) File "/home/jovyan/whisper/run.py", line 236, in init tokenizer_name == "gpt2" UnboundLocalError: local variable 'tokenizer_name' referenced before assignment

additional notes

Here's the fix #1958

@mrdrprofuroboros Would you mind changing the '==' to '=' please? We have fixed the typo and would update to github later.

mrdrprofuroboros commented 1 month ago

What do you mean? I’ve created a PR doing exactly this