Questions about gemini support in the spring ai module

[yangfeng20]

I want to use the gemini apiKey to access my gemini Ai instead of using project-id+location.

Currently, the spring ai module using gemini only supports project-id+location using google cloud. The apiKey mode in Google ai studio is not supported.

My request was to support the apiKey pattern, similar to openai.

[tzolov]

Hey @yangfeng20 could you provide us with a the Gemini documentation that explains how to create and use api keys?

[yangfeng20]

Ok, please refer to: https://ai.google.dev/gemini-api/docs/api-overview https://ai.google.dev/gemini-api/docs/api-key https://ai.google.dev/gemini-api/docs/get-started/rest

[vanduc2514]

@yangfeng20 There is a way to work around with the current spring-ai-vertex-ai-gemini dependency by replacing the low-level Request Formatter in HttpJsonPredictionServiceStub with the one pointing to the api endpoint of new Gemini AI

ProtoMessageRequestFormatter.<GenerateContentRequest>newBuilder()
                .setPath("/v1beta/models/{model=*}:generateContent", request -> Map.of("model", "gemini-pro"))
                .setQueryParamsExtractor(request -> Map.of("key", List.of("<GEMINI_API_KEY>")))
                .setRequestBodyExtractor(request -> ProtoRestSerializer.create().toBody("*", request.toBuilder().clearModel().build(),false))
                .build()

then set the VertexAI endpoint to https://generativelanguage.googleapis.com and use Transport.REST for VertexAI transport. The above code snippet only works for generateContent endpoint. For other endpoints to work, you will need to replace other request formatters. It's ok for the time being since Google has not provided an offical Java depdency yet.

[marcosamm]

@yangfeng20 There is a way to work around with the current spring-ai-vertex-ai-gemini dependency by replacing the low-level Request Formatter in HttpJsonPredictionServiceStub with the one pointing to the api endpoint of new Gemini AI

ProtoMessageRequestFormatter.<GenerateContentRequest>newBuilder()
                .setPath("/v1beta/models/{model=*}:generateContent", request -> Map.of("model", "gemini-pro"))
                .setQueryParamsExtractor(request -> Map.of("key", List.of("<GEMINI_API_KEY>")))
                .setRequestBodyExtractor(request -> ProtoRestSerializer.create().toBody("*", request.toBuilder().clearModel().build(),false))
                .build()

then set the VertexAI endpoint to https://generativelanguage.googleapis.com and use Transport.REST for VertexAI transport. The above code snippet only works for generateContent endpoint. For other endpoints to work, you will need to replace other request formatters. It's ok for the time being since Google has not provided an offical Java depdency yet.

@vanduc2514 , I didn't understand. Where do I need to set the transport to REST? Could you provide a short functional example using ProtoMessageRequestFormatter and setting Transport.REST?

[vanduc2514]

@marcosamm Sorry for making not clear. The new version of Spring AI 1.0.0-SNAPSHOT does not require the Transport.REST to be set. To implement this work-around you will need to create a custom implementation of PredictionServiceStub usingProtoMessageRequestFormatterthat points to the Gemini endpoint. Then use it for the VertexAiGeminiChatModel.

Here is an example for how to do it

https://gist.github.com/vanduc2514/0a74ec0ec160538c39313e73f27f8740

Then you will use the VertexAIAdapter class from the gist instead of the default VertexAI

@Bean
ChatModel chatModel() {
    var vertexAIAdapter = new VertexAIAdapter("gemini-base-url", "gemini-api-key")
    return new VertexAiGeminiChatModel(vertexAIAdapter);
}

You also need to disable VertexAiGeminiAutoConfiguration for skipping the auto bean creation.

[marcosamm]

@vanduc2514 , this works for me.

Sometimes I get timeout exceptions like this:

java.net.SocketTimeoutException: Read timed out
    at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[na:na]
    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[na:na]
    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[na:na]
    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[na:na]
    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1461) ~[na:na]
    at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1066) ~[na:na]
    at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:291) ~[na:na]
    at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:347) ~[na:na]
    at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420) ~[na:na]
    at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:399) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1690) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[na:na]
    at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:531) ~[na:na]
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:307) ~[na:na]
    at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[google-http-client-1.44.2.jar:1.44.2]
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[google-http-client-1.44.2.jar:1.44.2]
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[google-http-client-1.44.2.jar:1.44.2]
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[google-http-client-1.44.2.jar:1.44.2]
    at com.google.api.gax.httpjson.HttpRequestRunnable.run(HttpRequestRunnable.java:115) ~[gax-httpjson-2.49.0.jar:2.49.0]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

But it's normal in more complex interactions.

Thank you very much!

[vanduc2514]

@marcosamm You are welcome, I'm glad that it works for you. The issue you have seems related to SSL handshake timeout, does it frequently happen ?