dosubot[bot]
commented
2 hours ago Hey @HorseLuke! 👋 I'm here to help you with bug fixes, answer your questions, and guide you to become a contributor. Let's squash those bugs together!
I found two similar discussions that might be relevant to your issue:
- Adding a local ollama model error which was last updated on 2024-09-12.
- Adding a local ollama model error which was last updated on 2024-09-12.
To resolve the error "Input [TextPromptMessageContent(type=<PromptMessageContentType.TEXT: 'text'>, data='图片表述什么内容？'), ImagePromptMessageContent(type=<PromptMessageContentType.IMAGE: 'image'>, data='data:image/jpeg;base64,/9j/4AAQSkZJRgA...[ignore]...', detail=<DETAIL.LOW: 'low'>)] is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers" when using the Vision model "minicpm-v:8b-2.6-q4_K_M" with ollama in dify 0.8.3, you need to ensure that the prompt_messages are correctly formatted.
The _generate method in the OllamaLargeLanguageModel class expects the prompt_messages to be converted into a dictionary format that the API can understand. Specifically, the content of UserPromptMessage should be a string or a list/tuple of strings or integers.
Here is the relevant part of the code that handles this conversion:
def _convert_prompt_message_to_dict(self, message: PromptMessage) -> dict:
"""
Convert PromptMessage to dict for Ollama API
"""
if isinstance(message, UserPromptMessage):
message = cast(UserPromptMessage, message)
if isinstance(message.content, str):
message_dict = {"role": "user", "content": message.content}
else:
text = ""
images = []
for message_content in message.content:
if message_content.type == PromptMessageContentType.TEXT:
message_content = cast(TextPromptMessageContent, message_content)
text = message_content.data
elif message_content.type == PromptMessageContentType.IMAGE:
message_content = cast(ImagePromptMessageContent, message_content)
image_data = re.sub(r"^data:image\/[a-zA-Z]+;base64,", "", message_content.data)
images.append(image_data)
message_dict = {"role": "user", "content": text, "images": images}
elif isinstance(message, AssistantPromptMessage):
message = cast(AssistantPromptMessage, message)
message_dict = {"role": "assistant", "content": message.content}
elif isinstance(message, SystemPromptMessage):
message = cast(SystemPromptMessage, message)
message_dict = {"role": "system", "content": message.content}
else:
raise ValueError(f"Got unknown type {message}")
return message_dictEnsure that your prompt_messages are structured correctly before invoking the model. For example:
prompt_messages = [
UserPromptMessage(content=[
TextPromptMessageContent(type=PromptMessageContentType.TEXT, data='图片表述什么内容？'),
ImagePromptMessageContent(type=PromptMessageContentType.IMAGE, data='data:image/jpeg;base64,/9j/4AAQSkZJRgA...')
])
]This will be converted to:
{
"role": "user",
"content": "图片表述什么内容？",
"images": ["base64_encoded_image_data"]
}Make sure that the content field is a string or a list/tuple of strings or integers, and the images field contains the base64 encoded image data [1][2][3].
To continue talking to Dosu, mention @dosu.
HorseLuke
Self Checks
Dify version
0.8.3
Cloud or Self Hosted
Self Hosted (Docker)
Steps to reproduce
✔️ Expected Behavior
Chat will output normal from start to end.
❌ Actual Behavior
Chat output normal at first time, but then stop and clear output text, then it raised error at the top right of the page, saying:
Attach is from docker logs after setting DEBUG=true
dify-ollama-error-minicpm-v.zip