AzureChatOpenAI Streaming causes IndexError: list index out of range

[azohra]

System Info

langchain-0.0.205-py3, macos ventura, python 3.11

Who can help?

@hwchase17 / @agola11

Information

• [x] The official example notebooks/scripts https://python.langchain.com/docs/modules/model_io/models/chat/how_to/streaming

Related Components

• [X] LLMs/Chat Models

Reproduction

Reproduction code

# test.py
from langchain.chat_models import AzureChatOpenAI
from langchain.chat_models import ChatOpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.schema import (
    HumanMessage,
)

chat_1 = ChatOpenAI(streaming=True, 
                  callbacks=[StreamingStdOutCallbackHandler()],
                  openai_api_key="SOME-KEY",
                  model='gpt-3.5-turbo',
                  temperature=0.7, 
                  request_timeout=60,
                  max_retries=1)

chat_2 = AzureChatOpenAI(streaming=True, 
                       callbacks=[StreamingStdOutCallbackHandler()],
                       openai_api_base="https://some-org-openai.openai.azure.com/",
                       openai_api_version="2023-06-01-preview",
                       openai_api_key="SOME-KEY",
                       deployment_name='gpt-3_5',
                       temperature=0.7, 
                       request_timeout=60,
                       max_retries=1)

resp_1 = chat_1([HumanMessage(content="Write me a song about sparkling water.")])
resp_2 = chat_2([HumanMessage(content="Write me a song about sparkling water.")])

python test.py

Output of command 1 (OpenAI)

Verse 1:
Bubbles dancing in my cup
Refreshing taste, can't get enough
Clear and crisp, it's always there
A drink that's beyond compare

Chorus:
Sparkling water, oh how you shine
You make my taste buds come alive
With every sip, I feel so fine
Sparkling water, you're one of a kind

Verse 2:
A drink that's light and calorie-free
A healthier choice, it's plain to see
A perfect thirst quencher, day or night
With sparkling water, everything's right

Chorus:
Sparkling water, oh how you shine
You make my taste buds come alive
With every sip, I feel so fine
Sparkling water, you're one of a kind

Bridge:
From the fizzy sensation to the bubbles popping
You're the drink I never want to stop sipping
Whether at a party or on my own
Sparkling water, you're always in the zone

Chorus:
Sparkling water, oh how you shine
You make my taste buds come alive
With every sip, I feel so fine
Sparkling water, you're one of a kind

Outro:
Sparkling water, you're my go-to
A drink that always feels brand new
With each sip, I'm left in awe
Sparkling water, you're the perfect beverage 

Output of command 2 (Azure OpenAI)

raw.Traceback (most recent call last):
  File "/Users/someone/Development/test.py", line 29, in <module>
    resp_2 = chat_2([HumanMessage(content="Write me a song about sparkling water.")])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/langchain/chat_models/base.py", line 208, in __call__
    generation = self.generate(
                 ^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/langchain/chat_models/base.py", line 102, in generate
    raise e
  File "/opt/homebrew/lib/python3.11/site-packages/langchain/chat_models/base.py", line 94, in generate
    results = [
              ^
  File "/opt/homebrew/lib/python3.11/site-packages/langchain/chat_models/base.py", line 95, in <listcomp>
    self._generate(m, stop=stop, run_manager=run_manager, **kwargs)
  File "/opt/homebrew/lib/python3.11/site-packages/langchain/chat_models/openai.py", line 334, in _generate
    role = stream_resp["choices"][0]["delta"].get("role", role)
           ~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Expected behavior

I can't find anything in existing issues or documentation stating that there is a known bug in the AzureOpenAI Service Streaming.

[rock-you]

AzureChatOpenAI Streaming first response chunk:

FIX:

https://github.com/rock-you/langchain/commit/abd63a026cff1654e68ffcdd5a82828f87fc626b

[Sharvin26]

@rock-you are you planning to create PR for this? Because I am also facing same issue when using streaming with azure openai.

[giorgiococci]

I am also experiencing the same problem, and having this fix would be great

[jacoblee93]

Does this occur for you when setting version to 2023-05-15?

Looked into this on the JS side and it seems there's a change in events in the preview version - given that it's preview, it might be best to avoid making an unnecessary change here until the mainline version comes out:

https://github.com/Azure/azure-sdk-for-go/issues/21086

[PvanHengel]

Any ideas on a solution here, while I understand the comment about it being preview, it does seem like its here to stay, as far as I can see this is a blocker for being able to test out using OpenAI Functions with streaming responses.

[jacoblee93]

Unfortunately yeah - it's going to require some code changes. If you have time to submit a PR that'd be awesome, otherwise I can look tomorrow.

[JulyMood]

I had the same problem with version "2023-07-01-preview"

[PvanHengel]

Agreed yes I haven’t dug in to see what the fox would entail I can’t imagine it’s too difficult, but of course devil is in the details?

[jacobswe]

Opened up the fix above as a PR to hopefully help get this unblocked