search

spring-projects / spring-ai

An Application Framework for AI Engineering
https://docs.spring.io/spring-ai/reference/index.html
Apache License 2.0
3.35k stars 866 forks source link

Support URI in Media #1147

Open dmelikhov opened 3 months ago

dmelikhov commented 3 months ago

Context I can't attach GCS files when using Gemini:

new Media(MediaType.IMAGE_JPEG, new URL("https://storage.googleapis.com/my-bucket/my-image.jpeg"));
// io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Only GCS URIs are supported in file_uri and please make sure that the path is a valid GCS path.

new Media(MediaType.IMAGE_JPEG, new URL("gs://my-bucket/my-image.jpeg"));
// java.net.MalformedURLException: unknown protocol: gs

new Media(MediaType.IMAGE_JPEG, "gs://my-bucket/my-image.jpeg"));
// works, but this constructor is deprecated and marked for removal

Expected Behavior A constructor with URI parameter would solve the issue:

new Media(MediaType.IMAGE_JPEG, URI.create("gs://my-bucket/my-image.jpeg"));
ddobrin commented 1 month ago

Maybe you could test this approach extracted from a snippet: ... // bucket where image has been uploaded String imageURL = String.format("gs://%s/%s",bucketName, fileName);

    // create User Message for AI framework
    PromptTemplate userPromptTemplate = new PromptTemplate(imageUserMessage);
    Message imageAnalysisUserMessage = userPromptTemplate.createMessage(List.of(new 
                               Media(MimeTypeUtils.parseMimeType("image/*"), imageURL)));

...