Open ShaunDeng opened 1 month ago
Is this issue still reproducible when you use Spring Boot's autoconfiguration [3] (spring.ai.bedrock.aws.timeout=10m
) instead of creating the chat client yourself?
From my understanding, that constructor [1], which was introduced by https://github.com/spring-projects/spring-ai/pull/520 via 5c3ed1152a7a9318d431183b42f5ebf37a99ee90 , should've worked as expected [2].
Possibly related: https://github.com/aws/aws-sdk-java/issues/3072 and https://github.com/aws/aws-sdk-ruby/issues/2967
[1] https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/best-practices.html [2] https://github.com/spring-projects/spring-ai/blob/17c44237a51f85a30d128c21a335a7aee269f4c2/models/spring-ai-bedrock/src/main/java/org/springframework/ai/bedrock/api/AbstractBedrockApi.java#L151-L161 [3] https://docs.spring.io/spring-ai/reference/api/chat/bedrock/bedrock-anthropic3.html#_sample_controller
I tested it when I submitted it and it was ok. Have you ever tested using bean injection? @ShaunDeng
@impactCn , Thanks for your comments. Well when I use auto bean injection(spring-ai-bedrock-ai-spring-boot-starter), the timeout issue is not reproducible even with the default 5m timeout. But I need manually create(spring-ai-bedrock) those client/model instances by myself due to some purpose.
eh, I found another thing. When I directly use the api/client/model instance after manually created, it works fine. But when I created the api/client/model object, for example, Anthropic3ChatBedrockApi, I register it into spring application context, like below [1], and then get it back from spring application context to use it [2]. Then the issue is reproduced [1] `String beanName = IPluginsService.genDynamicAIChannelBeanName(channelId, modelCode); AWSBedrockLoginInfo awsLoginInfo = convertLoginInfo(loginInfo); Anthropic3ChatBedrockApi anthropicApi = new Anthropic3ChatBedrockApi( aiModelEnum.getRawCode(), StaticCredentialsProvider.create(AwsBasicCredentials.create(awsLoginInfo.getAccessKey(),awsLoginInfo.getSecretKey())), awsLoginInfo.getRegion(), JsonUtils.defaultObjectMapper(), timeout);
bean = applicationContext.getAutowireCapableBeanFactory().initializeBean(bean, beanName); DefaultListableBeanFactory beanFactory = (DefaultListableBeanFactory) applicationContext.getAutowireCapableBeanFactory(); beanFactory.registerSingleton(beanName, bean);`
[2]
Anthropic3ChatBedrockApi anthropicApi = beanRegistrar.getBean(IPluginsService.genDynamicAIChannelBeanName(channelId, modelCode), Anthropic3ChatBedrockApi.class); ... ... class beanRegistrar{ public <T> T getBean(String beanName, Class<T> type) { Object bean = context.getBean(beanName); if (type.isInstance(bean)) { return type.cast(bean); } return null; } }
Let me test it.
@impactCn Thank you, more info. I passed in a objectMapper, not sure if it's impacting, below is my object mapper:
JavaTimeModule javaTimeModule = new JavaTimeModule();
DateTimeFormatter dateTimeFormatter = DateTimeFormatter.ofPattern(TimeUtils.DATETIME_FORMAT_PATTERN); javaTimeModule.addSerializer(LocalDateTime.class, new LocalDateTimeSerializer(dateTimeFormatter)); javaTimeModule.addDeserializer(LocalDateTime.class, new LocalDateTimeDeserializer(dateTimeFormatter));
DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern(TimeUtils.DATE_FORMAT_PATTERN); javaTimeModule.addSerializer(LocalDate.class, new LocalDateSerializer(dateFormatter)); javaTimeModule.addDeserializer(LocalDate.class, new LocalDateDeserializer(dateFormatter));
javaTimeModule.addSerializer(OffsetDateTime.class, OffsetDateTimeSerializer.INSTANCE); objectMapper = new ObjectMapper() .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS) .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) .setSerializationInclusion(com.fasterxml.jackson.annotation.JsonInclude.Include.NON_NULL) .registerModule(javaTimeModule);
Bug description It seems like the timeout settings not working for class Anthropic3ChatBedrockApi,
Environment Spring AI 1.0.0-M1, Java 17,
Steps to reproduce 1 Set the read timeout to 10 minutes
IAWSService.AWSBedrockLoginInfo awsLoginInfo = convertLoginInfo(loginInfo); Anthropic3ChatBedrockApi anthropicApi = new Anthropic3ChatBedrockApi( aiModelEnum.getRawCode(), StaticCredentialsProvider.create(AwsBasicCredentials.create(awsLoginInfo.getAccessKey(),awsLoginInfo.getSecretKey())), awsLoginInfo.getRegion(), JsonUtils.defaultObjectMapper(), Duration.ofMinutes(10L));
2 Call with large contextBedrockAnthropic3ChatModel model = BedrockAnthropic3ChatModel(anthropicApi); model .call(new Prompt(messages));
3 Get the read timeout errors after about 2minutes 'Caused by: java.net.SocketTimeoutException: Read timed out at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288) at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966) at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484) at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478) at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70) at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069) at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254) at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104) at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231) at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228) at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:99) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:79) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:57) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:40) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:224) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) at software.amazon.awssdk.services.bedrockruntime.DefaultBedrockRuntimeClient.invokeModel(DefaultBedrockRuntimeClient.java:150) at org.springframework.ai.bedrock.api.AbstractBedrockApi.internalInvocation(AbstractBedrockApi.java:253) at org.springframework.ai.bedrock.anthropic3.api.Anthropic3ChatBedrockApi.chatCompletion(Anthropic3ChatBedrockApi.java:490) at org.springframework.ai.bedrock.anthropic3.BedrockAnthropic3ChatModel.call(BedrockAnthropic3ChatModel.java:79) at com.lenovo.tec.ai.codeg.gateway.plugins.impl.text.aws.AWSAnthropicChatPluginsServiceImpl.call(AWSAnthropicChatPluginsServiceImpl.java:110) at com.lenovo.tec.ai.codeg.gateway.plugins.impl.text.aws.AWSAnthropicChatPluginsServiceImpl.lambda$doChat$0(AWSAnthropicChatPluginsServiceImpl.java:96) at reactor.core.publisher.MonoCallable.call(MonoCallable.java:72) at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:228) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) '
Expected behavior The connection should not timeout until 10 minutes have passed since I set it to 10 minutes.
My Thinking Regarding AWS Java2 SDK best practices, they recommend setting both the apiCallAttemptTimeout and apiCallTimeout arguments (Please refer to the first picture below), but in Spring AI, only apiCallTimeout is set (Please refer to the second picture below). I'm not sure if this impacts. Picture 1
Picture 2
![image](https://github.com/user-attachments/assets/f813564b-f814-4ee1-8bc7-4dd2b1deff69)