search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.13k stars 4.53k forks source link

ChatCompletions between OpenAI and Azure.AI.OpenAI using gpt-35-turbo different #40752

Open mrunks opened 6 months ago

mrunks commented 6 months ago

Library name and version

Azure.AI.OpenAI 1.0.0-beta.11

Describe the bug

When I create a ChatCompletion with a ChatRequestSystemMessage and a ChatRequestUserMessage, the response is not the same as if I put the exact text in Azure AI Studio or if I put the same text in OpenAI's playground. If I use the Betalgo.OpenAI library to connect to OpenAi's api directly I receive the correct results. So it seems something is altering or not process the request correcting when I use the Azure OpenAI sdk to with gpt-35.

Expected behavior

I would expect each of the 4 options to return the same result for the Chat Completion.

Actual behavior

Using Azure OpenAI SDK, the answer is an empty string in certain questions and system prompts where as I receive the expected values using Azure Studio, OpenAi playground or OpenAI API directly.

Reproduction Steps

I will send the following text for a user and system prompt:

System Prompt:

You are given the following knowledge:

   knowledge:
   -----------------
    Identification X.X. Identification Product form : Mixture Product Name : SILVABRITE 100® Product code : A00000090 1.2. Recommended use and restrictions on use Recommended use : Alloys for brazing/soldering and other metallurgical processes 1.3. Supplier Lucas-Milhaupt, Inc. 5656 South Pennsylvania Ave. Cudahy, WI 53110 - USA T (414)-769-6000 LM_SDSinfo@lucasmilhaupt.com - www.Lucasmilhaupt.com 1.4. Emergency telephone number CHEMTREC within the USA and Canada: 1-800-424-9300 Emergency number : CHEMTREC outside the USA and Canada +1 701-741-5970 SECTION 2:       
  -----------------

user Prompt: What are the names of any products mentioned in the knowledge. If you cannot find any from the knowledge then say 'I don't know'.

Response from Azure Studio: "SILVABRITE 100®" Response from OpenAi Playground: "SILVABRITE 100®" Response from OpenAi Api: "SILVABRITE 100®"

Response from Azure.AI.OpenAI sdk ""

If I modify the text in system prompt I can get SILVABRITE 100 to appear but I would expect it to not need that change.

Environment

OS: Windows 11

Framework: Windows .NET 8.0

github-actions[bot] commented 6 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @trrwilson.

github-actions[bot] commented 6 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jpalvarezl @trrwilson.

mrunks commented 6 months ago

I wanted to add that I also experience Hallucinations or cached completions. I have some text that I send as the system part and keep getting back data that is not from the system text provided.

Yet when I uase Azure AI Studio it will return Not found if it cannot answer from the text I provide to the system message.

I would assume anything I am samping in Azure AI Studio should have the same results as the C# client correct ?