AzureOpenAI doesn't work with GPT 3.5 Turbo deployed models

[sergeyk93]

Looks like a similar issue as what OpenAI had when the model was introduced(arguments like best_of, logprobs weren't supported).

[XUWeijiang]

My solution is to subclass AzureOpenAPI (I also add stop parameter, as it is needed according to azure document.):

class NewAzureOpenAI(AzureOpenAI):
    stop: List[str] = None
    @property
    def _invocation_params(self):
        params = super()._invocation_params
        # fix InvalidRequestError: logprobs, best_of and echo parameters are not available on gpt-35-turbo model.
        params.pop('logprobs', None)
        params.pop('best_of', None)
        params.pop('echo', None)
        params['stop'] = self.stop
        return params

[sergeyk93]

Thanks @XUWeijiang. The problem with that approach is that you'll have to know in advance if the model is gpt 3.5, and the azure deployment name doesn't state that. I'm trying to build something more generic that can work with any set of params. In the OpenAI scenario, the framework handles it(even if we use the OpenAI wrapper), just not in the Azure scenario.

[aj-dev-smith]

It's unclear to me whether or not #1673 fixed this issue. From my tests it doesn't appear so. Is there any worked planned to address this issue at the moment?

[ekzhu]

@aj-dev-smith for chat models I think it's better to use AzureChatOpenAI instead. See the other comments in #1673 if that helps.

[ekzhu]

Added PR #1902 to improve AzureChatOpenAI.

[geg00]

https://python.langchain.com/en/latest/modules/chains/index_examples/summarize.html This link does not work if you attempt to use gpt-35-turbo

Now if you use the @XUWeijiang it is going to work but

I'm glad that @ekzhu is working on the AzureChatOpenAI solution but I don't see how this will help with the summarization of long documents.

[ekzhu]

Specifically what is the issue? Can you copy and paste the error?

On Mon, Mar 27, 2023 at 6:26 AM Gerardo Garcia @.***> wrote:

https://python.langchain.com/en/latest/modules/chains/index_examples/summarize.html This link does not work if you attempt to use gpt-35-turbo

Now if you use the @XUWeijiang https://github.com/XUWeijiang it is going to work but

I'm glad that @ekzhu https://github.com/ekzhu is working on the AzureChatOpenAI solution but I don't see how this will help with the summarization of long documents.

— Reply to this email directly, view it on GitHub https://github.com/hwchase17/langchain/issues/1591#issuecomment-1485089367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACOGLXUMPD6UKWODW6KYWDW6GIQJANCNFSM6AAAAAAVXCQOAM . You are receiving this because you were mentioned.Message ID: @.***>

[geg00]

I'm trying to mimic this page https://python.langchain.com/en/latest/modules/chains/index_examples/summarize.html

` import os import openai openai.api_type = "azure" openai.api_version = "2023-03-15-preview" openai.api_base = os.getenv("OPENAI_API_BASE") # Your Azure OpenAI resource's endpoint value . openai.api_key = os.getenv("OPENAI_API_KEY")

from langchain import PromptTemplate, LLMChain from langchain.llms import AzureOpenAI from langchain.text_splitter import CharacterTextSplitter from langchain.chains.mapreduce import MapReduceChain from langchain.prompts import PromptTemplate

llm = AzureOpenAI(temperature=0)

text_splitter = CharacterTextSplitter()

llm('Tell me a joke')

` This is the error message

` Output exceeds the size limit. Open the full output data in a text editor

InvalidRequestError Traceback (most recent call last) Cell In[13], line 1 ----> 1 llm('Tell me a joke')

File c:\Users\AppData\Local\mambaforge\lib\site-packages\langchain\llms\base.py:246, in BaseLLM.call(self, prompt, stop) 244 def call(self, prompt: str, stop: Optional[List[str]] = None) -> str: 245 """Check Cache and run the LLM on the given prompt and input.""" --> 246 return self.generate([prompt], stop=stop).generations[0][0].text

File c:\Users\AppData\Local\mambaforge\lib\site-packages\langchain\llms\base.py:140, in BaseLLM.generate(self, prompts, stop) 138 except (KeyboardInterrupt, Exception) as e: 139 self.callback_manager.on_llm_error(e, verbose=self.verbose) --> 140 raise e 141 self.callback_manager.on_llm_end(output, verbose=self.verbose) 142 return output

File c:\Users\AppData\Local\mambaforge\lib\site-packages\langchain\llms\base.py:137, in BaseLLM.generate(self, prompts, stop) 133 self.callback_manager.on_llm_start( 134 {"name": self.class.name}, prompts, verbose=self.verbose 135 ) 136 try: --> 137 output = self._generate(prompts, stop=stop) 138 except (KeyboardInterrupt, Exception) as e: 139 self.callback_manager.on_llm_error(e, verbose=self.verbose) ... 683 rbody, rcode, resp.data, rheaders, stream_error=stream_error 684 ) 685 return resp

InvalidRequestError: Resource not found even if I change the call to the llm to be llm = AzureOpenAI(deployment_id='gpt-35-turbo', temperature=0) ` I get the same error message.

Remember that we Azure you have to have a deployment_id Azure deploymentid cannot have '.' in them ¯_(ツ)/¯

And if you don't send any parameters to AzureOpenAI you will use text-davinci-003 which is defeats the principle of the using 3.5. I hope this helps.

[ekzhu]

This following code works for me:

from langchain.chat_models import AzureChatOpenAI
from langchain.text_splitter import CharacterTextSplitter
from langchain.schema import HumanMessage

llm = AzureChatOpenAI(
    deployment_name="gpt-35-turbo",
    temperature=0,
    openai_api_version="2023-03-15-preview",
)

text_splitter = CharacterTextSplitter()

print(llm([HumanMessage(content="Tell me a joke")]))

It is deployment_name not deployment_id, which probably explains the error message you got. Also AzureOpenAI class doesn't work with the chat completion API. So you need to use the AzureChatOpenAI class.

[ekzhu]

See this doc: https://python.langchain.com/en/latest/modules/models/chat/integrations/azure_chat_openai.html

[nanditamohan]

@ekzhu this was working for me Monday, but this morning, I see this issue using gpt-35-turbo, due to the same naming discrepancy between AOAI and OpenAI.... looks like Langchain may be checking for the decimal somewhere in the on_llm_end callback:

File "generator.py", line 282, in _get_citations_and_result
    result = chain(
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 116, in __call__
    raise e
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 113, in __call__
    outputs = self._call(inputs)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/combine_documents/base.py", line 56, in _call
    output, extra_return_dict = self.combine_docs(docs, **other_keys)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/combine_documents/stuff.py", line 89, in combine_docs
    return self.llm_chain.predict(**inputs), {}
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 151, in predict
    return self(kwargs)[self.output_key]
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 116, in __call__
    raise e
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 113, in __call__
    outputs = self._call(inputs)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 57, in _call
    return self.apply([inputs])[0]
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 118, in apply
    response = self.generate(input_list)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 62, in generate
    return self.llm.generate_prompt(prompts, stop)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chat_models/base.py", line 83, in generate_prompt
    self.callback_manager.on_llm_end(output, verbose=self.verbose)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/shared.py", line 47, in on_llm_end
    self._callback_manager.on_llm_end(response, **kwargs)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/base.py", line 159, in on_llm_end
    handler.on_llm_end(response)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/openai_info.py", line 78, in on_llm_end
    completion_cost = get_openai_model_cost_per_1k_tokens(
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/openai_info.py", line 39, in get_openai_model_cost_per_1k_tokens
    raise ValueError(
ValueError: Unknown model: gpt-35-turbo. Please provide a valid OpenAI model name.Known models are: gpt-4, gpt-4-0314, gpt-4-completion, gpt-4-0314-completion, gpt-4-32k, gpt-4-32k-0314, gpt-4-32k-completion, gpt-4-32k-0314-completion, gpt-3.5-turbo, gpt-3.5-turbo-0301, text-ada-001, ada, text-babbage-001, babbage, text-curie-001, curie, text-davinci-003, text-davinci-002, code-davinci-002

Do you know what the issue could be? I believe this is langchain 0.0.128

[ekzhu]

@ekzhu this was working for me Monday, but this morning, I see this issue using gpt-35-turbo, due to the same naming discrepancy between AOAI and OpenAI.... looks like Langchain may be checking for the decimal somewhere in the on_llm_end callback:

File "generator.py", line 282, in _get_citations_and_result
    result = chain(
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 116, in __call__
    raise e
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 113, in __call__
    outputs = self._call(inputs)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/combine_documents/base.py", line 56, in _call
    output, extra_return_dict = self.combine_docs(docs, **other_keys)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/combine_documents/stuff.py", line 89, in combine_docs
    return self.llm_chain.predict(**inputs), {}
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 151, in predict
    return self(kwargs)[self.output_key]
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 116, in __call__
    raise e
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/base.py", line 113, in __call__
    outputs = self._call(inputs)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 57, in _call
    return self.apply([inputs])[0]
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 118, in apply
    response = self.generate(input_list)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chains/llm.py", line 62, in generate
    return self.llm.generate_prompt(prompts, stop)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/chat_models/base.py", line 83, in generate_prompt
    self.callback_manager.on_llm_end(output, verbose=self.verbose)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/shared.py", line 47, in on_llm_end
    self._callback_manager.on_llm_end(response, **kwargs)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/base.py", line 159, in on_llm_end
    handler.on_llm_end(response)
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/openai_info.py", line 78, in on_llm_end
    completion_cost = get_openai_model_cost_per_1k_tokens(
  File "/azureml-envs/model-eval-env/lib/python3.8/site-packages/langchain/callbacks/openai_info.py", line 39, in get_openai_model_cost_per_1k_tokens
    raise ValueError(
ValueError: Unknown model: gpt-35-turbo. Please provide a valid OpenAI model name.Known models are: gpt-4, gpt-4-0314, gpt-4-completion, gpt-4-0314-completion, gpt-4-32k, gpt-4-32k-0314, gpt-4-32k-completion, gpt-4-32k-0314-completion, gpt-3.5-turbo, gpt-3.5-turbo-0301, text-ada-001, ada, text-babbage-001, babbage, text-curie-001, curie, text-davinci-003, text-davinci-002, code-davinci-002

Do you know what the issue could be? I believe this is langchain 0.0.128

For Azure Open AI the endpoint doesn't check for model name so you can just use one of the valid model name to pass langchain's validator. Just make sure you set deployment_name to match your deployment name on Azure Open AI portal.

[nanditamohan]

Ah I see - so when initializing the AzureChatOpenAI in langchain - I should set deployment_name to match AOAI deployment name, but model_name to match OpenAI model naming (with decimal point included)?

Wondering if it would be better if this translation should happen at the langchain validation layer, as I'd imagine AOAI customers assume model_name should be passed in to match whats in AOAI

[ekzhu]

Ah I see - so when initializing the AzureChatOpenAI in langchain - I should set deployment_name to match AOAI deployment name, but model_name to match OpenAI model naming (with decimal point included)?

yes.

Wondering if it would be better if this translation should happen at the langchain validation layer, as I'd imagine AOAI customers assume model_name should be passed in to match whats in AOAI

Right. You can submit a PR to fix this -- I know @hwchase17 is quite fast in responding to PR.

[ww1126811502]

Just use ChatOpenAI, because gpt-3.5 is a chat_model:

chat = ChatOpenAI(deployment_id=gptmodel,
                  model_kwargs={"api_key": openai.api_key, "api_base": openai.api_base, "api_type": openai.api_type,
                                "api_version": openai.api_version, },
                  model_name="gpt-35-turbo")

print(chat([HumanMessage(content="Translate this sentence from English to French. I love programming.")]))

[Joldnine]

Here is how I make it work with langchain==0.0.144:

        AzureChatOpenAI(
            temperature=0.01,
            deployment_name="gpt-35-turbo",
            openai_api_version="2023-03-15-preview",
        )

[szelesaron]

My issue was using ChatOpenAI instead of AzureChatOpenAI, thank you.

[dosubot[bot]]

Hi, @sergeyk93! I'm Dosu, and I'm helping the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale. Here's a summary of the current state of the issue:

The issue is related to AzureOpenAI not working with GPT 3.5 Turbo deployed models. A solution has been proposed by XUWeijiang to subclass AzureOpenAI and remove unsupported arguments. However, it seems that you are working on a more generic solution. In the comments, ekzhu suggests using AzureChatOpenAI instead and provides code that works for it. ekzhu has also added a pull request to improve AzureChatOpenAI. There are ongoing efforts to address the issue.

Now, we would like to know if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself. If we don't hear from you, the issue will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!