ChatDatabricks can't stream response: "KeyError: 'content'"

[mfeller2000]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_community.chat_models import ChatDatabricks

llm = ChatDatabricks(
    endpoint="my-endpoint",
    temperature=0.0,
)

for chunk in llm.stream("What is MLflow?"):
    print(chunk.content, end="|")

Error Message and Stack Trace (if applicable)

KeyError: 'content'
File <command-18425931933140>, line 8
      1 from langchain_community.chat_models import ChatDatabricks
      3 llm = ChatDatabricks(
      4     endpoint="my-endpoint",
      5     temperature=0.0,
      6 )
----> 8 for chunk in llm.stream("What is MLflow?"):
      9     print(chunk.content, end="|")
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-3b7ab85f-0c62-4fae-a71e-af61c05342b4/lib/python3.10/site-packages/langchain_community/chat_models/mlflow.py:161, in ChatMlflow.stream(self, input, config, stop, **kwargs)
    157     yield cast(
    158         BaseMessageChunk, self.invoke(input, config=config, stop=stop, **kwargs)
    159     )
    160 else:
--> 161     yield from super().stream(input, config, stop=stop, **kwargs)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-3b7ab85f-0c62-4fae-a71e-af61c05342b4/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:265, in BaseChatModel.stream(self, input, config, stop, **kwargs)
    258 except BaseException as e:
    259     run_manager.on_llm_error(
    260         e,
    261         response=LLMResult(
    262             generations=[[generation]] if generation else []
    263         ),
    264     )
--> 265     raise e
    266 else:
    267     run_manager.on_llm_end(LLMResult(generations=[[generation]]))
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-3b7ab85f-0c62-4fae-a71e-af61c05342b4/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py:245, in BaseChatModel.stream(self, input, config, stop, **kwargs)
    243 generation: Optional[ChatGenerationChunk] = None
    244 try:
--> 245     for chunk in self._stream(messages, stop=stop, **kwargs):
    246         if chunk.message.id is None:
    247             chunk.message.id = f"run-{run_manager.run_id}"
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-3b7ab85f-0c62-4fae-a71e-af61c05342b4/lib/python3.10/site-packages/langchain_community/chat_models/mlflow.py:184, in ChatMlflow._stream(self, messages, stop, run_manager, **kwargs)
    182 if first_chunk_role is None:
    183     first_chunk_role = chunk_delta.get("role")
--> 184 chunk = ChatMlflow._convert_delta_to_message_chunk(
    185     chunk_delta, first_chunk_role
    186 )
    188 generation_info = {}
    189 if finish_reason := choice.get("finish_reason"):
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-3b7ab85f-0c62-4fae-a71e-af61c05342b4/lib/python3.10/site-packages/langchain_community/chat_models/mlflow.py:239, in ChatMlflow._convert_delta_to_message_chunk(_dict, default_role)
    234 @staticmethod
    235 def _convert_delta_to_message_chunk(
    236     _dict: Mapping[str, Any], default_role: str
    237 ) -> BaseMessageChunk:
    238     role = _dict.get("role", default_role)
--> 239     content = _dict["content"]
    240     if role == "user":
    241         return HumanMessageChunk(content=content)

Description

I am trying to stream the response from the ChatDatabricks but this simply fails because it cannot find the 'content' key in the chunks. Also, the example code in the documentation does not work.

System Info

langchain==0.2.3 langchain-community==0.2.4 langchain-core==0.2.5 langchain-openai==0.1.8 langchain-text-splitters==0.2.1

[lucas-tucker]

I noticed that there is no langchain-databricks, and that the Databricks model is sourced from langchain_community, whereas other Chat models have all migrated to their own libraries (e.g. langchain-groq, langchain-huggingface)

[mfeller2000]

Update: This may be an endpoint issue on the Databricks side. After trying a different Databricks endpoint (Llama 3), the stream works fine. But trying with an external model (like GPT-4) will cause the above mentioned error. Invoking works fine though.