ray-project / llmperf

LLMPerf is a library for validating and benchmarking LLMs
Apache License 2.0
470 stars 69 forks source link

Added json mode prob #23

Closed kouroshHakha closed 1 week ago

kouroshHakha commented 6 months ago

This PR to set with what probability should queries get sent with a json mode response format. This feature allows us to benchmark clients under a mix of json mode and normal requests.