search

deepjavalibrary / djl-serving

A universal scalable machine learning model deployment solution
Apache License 2.0
182 stars 59 forks source link

[fix] Set tokenizer on output_formatter for TRT-LLM Handlers #2100

Closed maaquib closed 1 week ago

maaquib commented 1 week ago

Description

Fixes #2070

sindhuvahinis commented 1 week ago

Thanks Aaqib! Added a CI test https://github.com/deepjavalibrary/djl-serving/pull/2102 - I can test for you as well.

FYI, python of Tensorrt-llm does not support chat template now anyway, so skipped CI test for it.

maaquib commented 1 week ago

@sindhuvahinis Thanks. I ran some tests locally to verify this fixes the issue. Will run the CI once the PR is merged