[Question] When Azure will sync OpenAI ChatCompletion stream_options feature ?

SKYhuangjing commented 5 months ago

API Spec link

https://github.com/Azure/azure-rest-api-specs/blob/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/preview/2024-04-01-preview/inference.json

API Spec version

2024-04-01-preview

Question/Query

OpenAI ChatCompletion API has add stream_options {include_usage} for stream get usage, it will help developer to statistics the api cost, it's very good for us, so when azure will sync it?

Environment

REST API invoke it

zzhxiaofeng commented 5 months ago

@YunsongB Please help have a look, thank you.

dq7532183 commented 5 months ago

+1

dbg-raghulkrishna commented 5 months ago

+1

DmGilimovich commented 4 months ago

+1

janaka commented 4 months ago

Feat request here https://github.com/Azure/azure-sdk-for-net/issues/44237

SKYhuangjing commented 4 months ago

any update? @zzhxiaofeng @YunsongB

elabbarw commented 4 months ago

+1

supuwoerc commented 4 months ago

+1+1+1

pedro-n-rocha commented 4 months ago

+1

janaka commented 4 months ago

I'm guessing this all now comes along with the new OpenAI client that pulled in v2.0.x

dq7532183 commented 4 months ago

when?

janaka commented 4 months ago

https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/openai/Azure.AI.OpenAI/CHANGELOG.md#200-beta3-unreleased

sandangel commented 3 months ago

Hi, I tried with 2024-05-01-preview but still get the error. 400 bad request. I think it needs to be enabled on the server side, not client.

sergiogarlez commented 3 months ago

+1

janaka commented 3 months ago

Even in the new OpenAI lib (That Azure OpenAI v2.0.* depends on), this isn't implemented yet.

See https://github.com/openai/openai-dotnet/issues/103#issuecomment-2201442467

mfournioux commented 3 months ago

Are there any update on this issue?

Funnyguy77 commented 2 months ago

Bump. Hoping to see this implemented.

xin9le commented 2 months ago

It appears that the OpenAI API is prepared to accommodate token usage reporting, as evidenced by the IncludeUsage = true setting in the Azure.AI.OpenAI v2 SDK implementation. While the specific obstacles preventing REST API support remain unclear, there is likely a substantial number of developers who desire to obtain token usage information via the Streaming API. Given that the current situation necessitates manual calculation of token usage, which is a considerable inconvenience, we earnestly hope for prompt support of this feature.

Funnyguy77 commented 2 months ago

Given that the current situation necessitates manual calculation of token usage, which is a considerable inconvenience, we earnestly hope for prompt support of this feature.

I would also like to note that manual token calculation is often not accurate, in addition to being a large inconvenience. Making this feature even more desirable to developers, like myself.

DrShushen commented 2 months ago

This would be very useful - as @Funnyguy77 said, counting tokens manually is very error prone, and not maintainable.

janaka commented 2 months ago

It would be great if the Azure OpenAI team could give customers some indication as to when this feature will be released.

In general you really need to do a better job communicating the policy for maintaining feature parity with OpenAI feature releases. Right not this is one of several inference features that many are waiting on with absolutely no info on the Azure OpenAI teams intentions.

MartialBE commented 2 months ago

When I try to request gpt-4o-mini, stream_options is valid for me but not for other models. Why is this inconsistency happening?

SKYhuangjing commented 1 month ago

https://github.com/Azure/azure-rest-api-specs/pull/30087 it seems have some update, but not nofity us.

SKYhuangjing commented 1 month ago

When I try to request gpt-4o-mini, stream_options is valid for me but not for other models. Why is this inconsistency happening?

Compare eastus & northcentralus API Result (api_version = 2024-07-01-preview)

eastus has support stream_options
northcentralus has not support it hard to see why...

janaka commented 1 month ago

When I try to request gpt-4o-mini, stream_options is valid for me but not for other models. Why is this inconsistency happening?

Compare eastus & northcentralus API Result (api_version = 2024-07-01-preview)

eastus has support stream_options

northcentralus has not support it hard to see why...

iirc for another feature (tool_choice=required) somebody reported similar that it's avail in one region but not another. I think with that it was avail in one of the euro region but not in eastus.

SoulBlaze4j commented 1 month ago

How is it going? Can't imagine it has been a long long time and Azure still cannot catch up with this feature even in the preview API version! :))

MartialBE commented 2 weeks ago

I'm currently in the swedencentral/northcentralus/japaneast regions, and stream_options is working correctly. Is it fully available now? api-version: 2024-07-01-preview

SKYhuangjing commented 1 week ago

has been released https://learn.microsoft.com/zh-cn/azure/ai-services/openai/api-version-deprecation#changes-between-2024-09-01-preview-and-2024-08-01-preview

SoulBlaze4j commented 1 week ago

has been released learn.microsoft.com/zh-cn/azure/ai-services/openai/api-version-deprecation#changes-between-2024-09-01-preview-and-2024-08-01-preview

GPT-4 vision-preview still does not support this feature, I tested in westus region and it returned { "error": { "message": "1 validation error for Request\nbody -> stream_options\n extra fields not permitted (type=value_error.extra)", "type": "invalid_request_error", "param": null, "code": null } }

Azure / azure-rest-api-specs