TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
Hi! I'm wondering if there's a straightforward way to kill/clean an in-progress decoding from a GenerationSession? My use case is running VAD + Whisper in parallel and early-stopping the decoding if VAD marks the chunk as silent.
Hi! I'm wondering if there's a straightforward way to kill/clean an in-progress decoding from a
GenerationSession? My use case is running VAD + Whisper in parallel and early-stopping the decoding if VAD marks the chunk as silent.