[Bug]: Cohere streaming fails silently

[krrishdholakia]

What happened?

import os
os.environ["COHERE_API_KEY"] = "bad-key"

from litellm import completion 

try:
  completion(model="command-nightly", messages=[{"role": "user", "content": "Hey"}], stream=True)
except Exception as e: 
  print(e)

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

Investigating this further. This is caused by our current implementation of streaming which try-excepts, and if it's an exception assumes the stream is exhausted and returned an empty response object.

However, this means that errors silently fail across all providers. Will need to actually find the finish reason raised and stop when that occurs.

If it's an exception have it enter our exception mapping logic.

[krrishdholakia]

Impacted providers:

• [x] Cohere
• [x] Anthropic
• [x] Replicate
• [x] TogetherAI
• [x] Huggingface
• [x] VertexAI
• [x] AI21
• [x] Aleph Alpha
• [x] NLP Cloud
• [x] Bedrock
• [x] Sagemaker

[krrishdholakia]

Pushed in - 889679a0dd072760f80db043503d45b484062050