Always Flush UIDs after `GeneratorReply`

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

Apache License 2.0

1.91k stars 175 forks source link

Closed weiqisun closed 5 months ago

weiqisun commented 5 months ago

This PR fixes the issue where the MII server hangs after encountering 9999 exceptions. This issue was initially reported in https://github.com/microsoft/DeepSpeed-MII/issues/489.

weiqisun commented 5 months ago

NVM, this fix doesn't solve the issue since the uid is never returned after the exception in put_request