search

deepjavalibrary / djl-serving

A universal scalable machine learning model deployment solution
Apache License 2.0
182 stars 59 forks source link

[vllm, lmi-dist] add support for top_n_tokens #2051

Closed sindhuvahinis closed 2 weeks ago

sindhuvahinis commented 3 weeks ago

Description

Supported top_n_token for vllm, lmi-dist

ToDo:

  1. Add it chat output_formatter
  2. Add parameters in integration test
  3. Adding unit test cases with fake rolling batch