search

timoklimmer / powerproxy-aoai

Monitors and processes traffic to and from Azure OpenAI endpoints.
MIT License
83 stars 23 forks source link

Feature: Configure HTTPX timeouts in config & Error recovery #82

Closed codylittle closed 2 weeks ago

codylittle commented 1 month ago

We've recently been receiving a high number of httpx.ConnectTimeout exceptions on our PAYG endpoints. The ability to configure the timeouts within the config as well as recover from any exceptions within aoai_targets loop and try the next endpoint would be great too.

Version: v0.10.1

timoklimmer commented 1 month ago

Hi @codylittle, I am a bit hesitant to eat exceptions because they indicate that something is wrong and should be fixed. I need to think about it. Regarding the connect timeouts: I have increased the default connect timeout to 15 seconds now and released another version. If you still need configurable timeouts, please let me know.

codylittle commented 1 month ago

@timoklimmer thanks for raising those. I understand hesitations about continuing on exceptions, but I do believe that having at least read timeouts optionally configurable with default values would be beneficial. We've had some PAYG 100k tokens prompt 2k tokens complete inferences go upwards of 2 minutes and in our org's fork we have them hard coded to 5 minutes but I'd have to assume others would be experiencing similar issues with the defaults given the poor performance of PAYG. (We have data sovereignty requirements, so need to suck up the PAYG latency even during peak usage)

timoklimmer commented 1 month ago

Ok, I will add a configuration option for timeouts then. Seems like your problem is more complex than I thought initially.

timoklimmer commented 2 weeks ago

Hey @codylittle, it took a while, but PowerProxy has an option to configure custom limits and timeouts for the connection to AOAI now. Enjoy 🎉 I will close the issue then. Again and as always, thanks for bringing this up.