For best_of, we wait for the whole generation is finished, because we would not know which of the multiple sequences has the highest cumulative probability until the end of the generation. This is the reason, TGI supports best_of only during non streaming case.
This PR, introduces best_of support in json_output_formatter.
Unit test case:
Added unit test case to validate the output formatter.
Integration tests:
In order to test this in integration test, vllm/lmi-dist handler should support this. Will create a new PR for this, as the changes are a lot. Meanwhile, this PR changes should not fail any integration tests.
Description
For best_of, we wait for the whole generation is finished, because we would not know which of the multiple sequences has the highest cumulative probability until the end of the generation. This is the reason, TGI supports best_of only during non streaming case.
This PR, introduces best_of support in json_output_formatter.
Unit test case: Added unit test case to validate the output formatter.
Integration tests: In order to test this in integration test, vllm/lmi-dist handler should support this. Will create a new PR for this, as the changes are a lot. Meanwhile, this PR changes should not fail any integration tests.