ZhiPuAiChatModel createRequest bug

[lwgCodePlus]

If the message type used is UserMessage, it will not be built correctly

List<ChatCompletionMessage> chatCompletionMessages = prompt.getInstructions().stream().map(m -> {
            // Add text content.
            List<MediaContent> contents = new ArrayList<>(List.of(new MediaContent(m.getContent())));
            if (m instanceof UserMessage userMessage) {
                if (!CollectionUtils.isEmpty(userMessage.getMedia())) {
                    // Add media content.
                    contents.addAll(userMessage.getMedia()
                        .stream()
                        .map(media -> new MediaContent(
                                new MediaContent.ImageUrl(this.fromMediaData(media.getMimeType(), media.getData()))))
                        .toList());
                }
            }

            return new ChatCompletionMessage(contents, ChatCompletionMessage.Role.valueOf(m.getMessageType().name()));
        }).toList();

Here's my modified example

List<ChatCompletionMessage> chatCompletionMessages = prompt.getInstructions().stream().map(m -> {
                    if (m instanceof UserMessage userMessage) {
                        List<ChatCompletionMessage.MediaContent> contents = userMessage.getMedia().stream()
                                .flatMap(media -> Stream.of(
                                        new ChatCompletionMessage.MediaContent(userMessage.getContent()),
                                        new ChatCompletionMessage.MediaContent(
                                                new MediaContent.ImageUrl(this.fromMediaData(media.getMimeType(), media.getData())))
                                ))
                                .collect(Collectors.toList());

                        if (!contents.isEmpty()) {
                            return new ChatCompletionMessage(contents, ChatCompletionMessage.Role.valueOf(m.getMessageType().name()));
                        }
                    }
                    return new ChatCompletionMessage(m.getContent(), ChatCompletionMessage.Role.valueOf(m.getMessageType().name()));
                }).collect(Collectors.toList());

[lwgCodePlus]

@mxsl-gr

[magicgone-cn]

I find that only model glm-4-alltools support the content structure below "content": [ { "type":"text", "text":"hello I'm Mike, 6 years old" }] .

for example, if I send a request like this, other models like glm-4-air will only say hello to me, not tell me correct age. if I use model glm-4-alltools, it will tell me "You will be 7 years old next year"

completion = client.chat.completions.create(
    model="glm-4-alltools",
    messages=[
        {
            "role": "user",
            "content": [{
                "type": "text",
                "text": "I'm Mike, 6 years old"
            }]
        },
        {
            "role": "assistant",
            "content": [{
                "type": "text",
                "text": "Hello Mike"
            }]
        },
        {
            "role": "user",
            "content": [{
                "type": "text",
                "text": "Do you know how old am I next year"
            }]
        }
    ],
    temperature=0.7,
    stream=True
)

[mxsl-gr]

@mxsl-gr

hi, @lwgCodePlus sorry, i went on a trip and just back now. but i didn't understand this problem. is there any example or error message?

[mxsl-gr]

I find that only model glm-4-alltools support the content structure below我发现只有模型glm-4-alltools支持下面的内容结构 "content": [ { "type":"text", "text":"hello I'm Mike, 6 years old" }] .“content”： [ { “type”：“text”， “text”：“你好，我是Mike，6岁” }] .

for example, if I send a request like this, other models like glm-4-air will only say hello to me, not tell me correct age.例如，如果我发送这样的请求，其他型号（如GLM-4-Air）只会你好，不会告诉我正确的年龄。 if I use model glm-4-alltools, it will tell me "You will be 7 years old next year"如果我使用模型 glm-4-alltools，它会告诉我“明年你就 7 岁了”

completion = client.chat.completions.create(
    model="glm-4-alltools",
    messages=[
        {
            "role": "user",
            "content": [{
                "type": "text",
                "text": "I'm Mike, 6 years old"
            }]
        },
        {
            "role": "assistant",
            "content": [{
                "type": "text",
                "text": "Hello Mike"
            }]
        },
        {
            "role": "user",
            "content": [{
                "type": "text",
                "text": "Do you know how old am I next year"
            }]
        }
    ],
    temperature=0.7,
    stream=True
)

hi, @magicgone-cn , i tried and indeed there is a problem. use model glm-4-air and {content:{type:'', text:''}}, it seems that only the first user message is taking effect, and other messages are being ignored. i checked the ZhiPu document, the messages parameter between glm-4 and glm-4-alltools is inconsistent, even though they are using seem the same api https://open.bigmodel.cn/api/paas/v4/chat/completions. i think there is already a PR inadvertently fix it, using string content when user message is no media content.

[markpollack]

Does the current PR fix the issues observed in this conversation?

https://github.com/spring-projects/spring-ai/pull/1188