[QUERY]ERROR com.azure.core.http.netty.NettyAsyncHttpClient - java.util.concurrent.TimeoutException: Channel response timed out after 60000 milliseconds.

[PengMai1998]

Query/Question I am calling the Azure OpenAI API in Java, I am using version 1.0.0-beta.5.

OpenAIClient client = new OpenAIClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();

IterableStream<ChatCompletions> chatCompletionsStream = client.getChatCompletionsStream(deploymentOrModelId, new ChatCompletionsOptions(chatMessages));

I have noticed that if there is a period of inactivity (around 10 minutes) without using the API, the next time I try to use the API, it will result in a timeout error. However, after the timeout, subsequent API calls work normally. I am puzzled by this phenomenon and would appreciate the help of an expert to resolve it. Here is the error message:

2023-11-22 15:14:25.358 [reactor-http-epoll-73] WARN  reactor.netty.http.client.HttpClientConnect - [cea21651-2, L:/172.24.242.8:50746 - R:public-key.openai.azure.com/20.232.91.180:443] The connection observed an error
java.util.concurrent.TimeoutException: Channel response timed out after 60000 milliseconds.
    at com.azure.core.http.netty.implementation.AzureSdkHandler.responseTimedOut(AzureSdkHandler.java:200)
    at com.azure.core.http.netty.implementation.AzureSdkHandler.lambda$startResponseTracking$2(AzureSdkHandler.java:184)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:394)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:750)
2023-11-22 15:14:25.359 [http-nio-8443-exec-5] ERROR com.azure.core.http.netty.NettyAsyncHttpClient - java.util.concurrent.TimeoutException: Channel response timed out after 60000 milliseconds.

Thank you very much.

[joshfree]

@mssfang could you please assist @PengMai1998 with the Azure OpenAI issue above?

[mssfang]

Hi @PengMai1998 thank you for reaching out to us.

Netty (also other Http Clients, such as OkHttp, JDK HttpClient, and Vertx) supports close the connection and throwing an exception if the connection is idle for a while. It is an expected behavior. Also, 10-mins idle time without any activity is really long time. May I know how do you use the getChatCompletionsStream API? keeping a reference for iterating the stream response?

[github-actions[bot]]

Hi @PengMai1998. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @PengMai1998, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[PengMai1998]

Hi @mssfang , Thank you very much for your response. I apologize for replying so late. I am using getChatCompletionsStream in the Controller of my Spring Boot application. Once the backend receives a request from the frontend, I make a request to Azure OpenAI. Here is my code for your reference:

    @PostMapping(path="/chat/{flag}/{ids}")
    public Result startChat(@PathVariable("flag") Integer flag, @PathVariable("ids") List<Integer> ids, @RequestBody String question) throws Exception {
        question = URLDecoder.decode(question.substring(0, question.lastIndexOf('=')));
        User loginUser = UserUtil.getLoginUser();
        String username = loginUser.getUsername() + loginUser.getName();
        SseEmitter sseEmitter = SseEmitterUtil.getUser(username);
        String azureOpenaiKey = "**************************";
        String endpoint = "***************";
        String deploymentOrModelId = "***************";
        Integer numUse = 1;
        OpenAIClient client = new OpenAIClientBuilder()
                .endpoint(endpoint)
                .credential(new AzureKeyCredential(azureOpenaiKey))
                .buildClient();
        List<ChatMessage> chatMessages = new ArrayList<>();
        chatMessages.add(new ChatMessage(ChatRole.USER, question));
        DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String requestTime = dateFormat.format(new Date());
        StringBuilder msg = new StringBuilder();
        try {
            IterableStream<ChatCompletions> chatCompletionsStream = client.getChatCompletionsStream(deploymentOrModelId, new ChatCompletionsOptions(chatMessages));
            chatCompletionsStream.stream().skip(1).forEach(chatCompletions -> {
                ChatMessage delta = chatCompletions.getChoices().get(0).getDelta();
                if (delta.getRole() != null) {
                    System.out.println("Role = " + delta.getRole());
                }
                if (delta.getContent() != null) {
                    try {
                        String data = delta.getContent().replaceAll(" ", "&#32;").replaceAll("\\n", "&#92n");
                        Thread.sleep(50);
                        sseEmitter.send(data);
                        msg.append(delta.getContent());
                    } catch (IOException e) {
                        LOGGER.error("[用户]：" + loginUser.getId() + "-" + loginUser.getName() + "-" + "使用CHATGPT时发生了错误");
                        SseEmitterUtil.closeSseConnect(username);
                    } catch (InterruptedException e) {
                        throw new RuntimeException(e);
                    }
                }
            });
        } catch (Exception e) {
            try {
                JSONObject jsonObject = JSON.parseObject(e.getMessage().substring(e.getMessage().indexOf('"')+1 , e.getMessage().lastIndexOf('"')));
                JSONObject errorObject = jsonObject.getJSONObject("error");
                Map<String, String> Error = new HashMap<>();
                Error.put("error", errorObject.getString("message"));
                sseEmitter.send(Error);
                sseEmitter.send("[DONE]");
            } catch (IOException ex) {
                SseEmitterUtil.closeSseConnect(username);
                ex.printStackTrace();
            } finally {
                SseEmitterUtil.closeSseConnect(username);
                return new Result(Code.SEND_ERR, null, "连接被OpenAI关闭");
            }
        }
        if(SseEmitterUtil.getUser(username) == null) {
            return new Result(Code.SEND_ERR, null, "连接被OpenAI关闭");
        }
        // 回答完成，可以将提问和回答都存入数据库备份
        // 存储提问
        ChatGPTRecord chatGPTRecordRequest = new ChatGPTRecord();
        chatGPTRecordRequest.setContext(question);
        chatGPTRecordRequest.setTime(requestTime);
        chatGPTRecordRequest.setUid(loginUser.getId());
        chatGPTRecordRequest.setContextType("Request");
        // 存储回答
        String responseTime = dateFormat.format(new Date());
        ChatGPTRecord chatGPTRecordResponse = new ChatGPTRecord();
        chatGPTRecordResponse.setContext(msg.toString());
        chatGPTRecordResponse.setTime(responseTime);
        chatGPTRecordResponse.setUid(loginUser.getId());
        chatGPTRecordResponse.setContextType("Response");
        chatGPTService.saveChatGPTRecordAndRefreshUsage(chatGPTRecordRequest, chatGPTRecordResponse, numUse);
        try {
            sseEmitter.send("[DONE]");
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            SseEmitterUtil.closeSseConnect(username);
        }
        SseEmitterUtil.closeSseConnect(username);
        return new Result(Code.GET_OK, null, "OK");
    }

When the API is not used for a period of time, using it again will definitely result in a timeout exception.

[banyan-god]

I believe it is timing out in one minute, how do configure this timeout

[mssfang]

@banyan-god Yeah. it is default to 60 seconds.

@PengMai1998 Can you try to configure the Netty HTTP Client with a longer max idle time when creating it.

         ConnectionProvider connectionProvider = ConnectionProvider.builder("myHttpConnection")
                .maxConnections(500)
                .pendingAcquireMaxCount(-1)
                .maxIdleTime(Duration.ofMinutes(30)) // Configure the max idle time here.
                .build();

        HttpClient nettyClient = new NettyAsyncHttpClientBuilder()
                .connectionProvider(connectionProvider)
                .build();

        OpenAIClient client = new OpenAIClientBuilder()
            .endpoint(endpoint)
            .httpClient(nettyClient)
            .credential(credential)
            .buildClient();

[github-actions[bot]]

Hi @PengMai1998. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[github-actions[bot]]

Hi @PengMai1998, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!