search

Azure / aoai-smart-loadbalancing

Smart load balancing for Azure OpenAI endpoints
MIT License
75 stars 26 forks source link

How can we Handle the timeout for the HttpForwarded for models requires more time to respond #6

Closed jayendranarumugam closed 7 months ago

jayendranarumugam commented 7 months ago

I tried out the solution looks like some of the models require more time to respond, so we need to override the default timeout at HttpForwarder. During the timeout I got the error like below

  Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      RequestTimedOut: The request timed out before receiving a response.
      System.Threading.Tasks.TaskCanceledException: The operation was canceled.
         at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__88_0(Object s, CancellationToken cancellationToken)
         at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
         at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
         at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
         at System.Threading.ThreadPoolWorkQueue.Dispatch()
         at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
         at System.Threading.Thread.StartCallback()
      --- End of stack trace from previous location ---
         at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
         at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)
warn: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      RequestTimedOut: The request timed out before receiving a response.
      System.Threading.Tasks.TaskCanceledException: The operation was canceled.
         at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__88_0(Object s, CancellationToken cancellationToken)
         at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
         at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
         at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
         at System.Threading.TimerQueue.FireNextTimers()
         at System.Threading.ThreadPoolWorkQueue.Dispatch()
         at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
         at System.Threading.Thread.StartCallback()
      --- End of stack trace from previous location ---
         at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
         at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at Yarp.ReverseProxy.Forwarder.HttpForwarder.SendAsync(HttpContext context, String destinationPrefix, HttpMessageInvoker httpClient, ForwarderRequestConfig requestConfig, HttpTransformer transformer, CancellationToken cancellationToken)
warn: Yarp.ReverseProxy.Forwarder.HttpForwarder[48]
      RequestTimedOut: The request timed out before receiving a response.
andredewes commented 7 months ago

Now we have the HTTP_TIMEOUT_SECONDS as one optional environment variable that you can set. Please check the README