This PR updates the benchmark with the following changes:
Fix the preprocessing of ShareGPT dataset. In ShareGPT dataset, sometimes a conversation entry starts with role GPT instead of role human, but previously the benchmark always selects the first message as the input and second as output. Therefore it might cause sending the GPT output in the dataset as a request input to benchmark servers. This does not align with the expectation and we thus fix it.
We make sure that for a given seed, given tokenizer and a given number of requests, the sampled requests are always the same across different runs. This improves the determinism and reproducibility.
Introduce logging raw request records into file. The raw request records include all information of a request, including its input prompt, measured metrics, error message, etc.
This PR updates the benchmark with the following changes:
Fix the preprocessing of ShareGPT dataset. In ShareGPT dataset, sometimes a conversation entry starts with role GPT instead of role human, but previously the benchmark always selects the first message as the input and second as output. Therefore it might cause sending the GPT output in the dataset as a request input to benchmark servers. This does not align with the expectation and we thus fix it.
We make sure that for a given seed, given tokenizer and a given number of requests, the sampled requests are always the same across different runs. This improves the determinism and reproducibility.
Introduce logging raw request records into file. The raw request records include all information of a request, including its input prompt, measured metrics, error message, etc.