Can tensorrtllm backend support LogitsProcessor?

triton-inference-server / tensorrtllm_backend

The Triton TensorRT-LLM Backend

Apache License 2.0

588 stars 81 forks source link

Can tensorrtllm backend support LogitsProcessor? #391

Open Muxv opened 2 months ago

Muxv commented 2 months ago

In TensorRT-LLM, it is possible to integrate a LogitsProcessor during model inference to control the behavior of the inference process. Is it feasible to add a similar interface in the tensorrtllm backend to implement LogitsProcessor?

byshiue commented 2 months ago

Currently, TRT LLM backend does not support such requirement.