fix: subsequent requests cannot be sent until 'num_concurrent_requests' requests have all finished in non-block mode

ray-project / llmperf

LLMPerf is a library for validating and benchmarking LLMs

Apache License 2.0

659 stars 113 forks source link

Closed llsj14 closed 5 months ago

llsj14 commented 5 months ago

subsequent requests cannot be sent until whole requests have all finished even in non-block mode.
get_next_ready function should be fixed to return its result as soon as possible in non-block mode.

llsj14 commented 5 months ago

This commit is not enough to fix the problem, so close this PR.