[FEATURE REQ] Azure.AI.OpenAI: Support HuggingFace chat completion streaming API

[dluc]

Library name

Azure.AI.OpenAI

Please describe the feature.

HuggingFace chat completion streaming API is designed to imitate OpenAI streaming response. However, due to a couple of minor differences, when pointing Azure SDK OpenAIClient to HuggingFace, method GetCompletionsStreaming hangs indefinitely:

1. HF doesn't terminate a stream with [DONE], so SseAsyncEnumerator while loop never breaks.

2. HF doesn't support NucleusSamplingFactor 0.0, and returns an error {"error":"Input validation error: `top_p` must be > 0.0 and < 1.0","error_type":"validation"}. Unfortunately the response status code is 200 OK so it doesn't trigger any exception. Users could workaround this issue by passing 0.01 instead, but there's no exception suggesting to change the value.

It would be great if Azure AI SDK had a way to workaround these issues, for instance:

• Detect non-deserializable responses, e.g. SseAsyncEnumerator<Completions> should throw an exception when deserializing {"error":"Input validation error: `top_p` must be > 0.0 and < 1.0","error_type":"validation"}.
• Detect when the remote endpoint stops sending data - as far as I know the HTTP connection is closed, so the client could stop waiting, without reaching a Task timeout

See also https://github.com/huggingface/text-generation-inference/issues/1896 and https://github.com/microsoft/kernel-memory/issues/388

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @trrwilson.