spring-projects / spring-ai

An Application Framework for AI Engineering
https://docs.spring.io/spring-ai/reference/index.html
Apache License 2.0
3.2k stars 811 forks source link

has error multiple modal calling in zhipuAi #854

Closed viewhang closed 4 months ago

viewhang commented 4 months ago

Bug description zhipu multiple modal api doc

image

The image_url. url here can be a base64 string but should not include the prefix like this data:image/png;base64,

image

Environment Spring AI version:

 <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-zhipuai-spring-boot-starter</artifactId>
            <version>1.0.0-SNAPSHOT</version>
        </dependency>

Java version:17

Steps to reproduce

final Media recourceMedia = new Media(MimeTypeUtils.IMAGE_PNG, new FileSystemResource(imageFile));
        String content = chatClient.prompt()
                .user(userSpec -> userSpec
                        .text(message)
                        .media(recourceMedia)
                )
                .call()
                .content();

return

org.springframework.ai.retry.NonTransientAiException: 400 - {"error":{"code":"1210","message":"图片输入格式/解析错误"}}

Expected behavior request success

Minimal Complete Reproducible example see Steps to reproduce

viewhang commented 4 months ago

success like this,but @Deprecated(since = "1.0.0 M1", forRemoval = true)

 String chatWithImage(String message, File imageFile) throws IOException {
        final String base64Str = Base64.getEncoder().encodeToString(Files.readAllBytes(Paths.get(imageFile.getPath())));
        final Media recourceMedia = new Media(MimeTypeUtils.IMAGE_PNG, base64Str);
        String content = chatClient.prompt()
                .user(userSpec -> userSpec
                        .text(message)
                        .media(recourceMedia)
                )
                .call()
                .content();
        logger.info(content);
        return content;
    }
mxsl-gr commented 4 months ago

hi, I haven't added multi-modal (GLM-4V) support yet. i will take a look at this issue later.

mxsl-gr commented 4 months ago

i have add multi-model support in PR #859

mxsl-gr commented 4 months ago

@viewhang hi, i noticed the PR has been merged, you can pull the code and give it try 😊

viewhang commented 4 months ago

it works well Thanks