search

timoklimmer / powerproxy-aoai

Monitors and processes traffic to and from Azure OpenAI endpoints.
MIT License
85 stars 24 forks source link

Update powerproxy.py: httpx client default timeout is only 5 sec, bump to 120 sec to get non-streaming responses taking longer through #36

Closed Johann-Foerster closed 7 months ago

Johann-Foerster commented 7 months ago

120 sec timeout chosen as used elsewhere throughout the application

timoklimmer commented 7 months ago

@Johann-Foerster Thanks for sharing your PR. We need to distinguish here between the different types of timeout. We can increase the read timeout as you suggested, but we should not increase all timeouts in general. As example, with a high connect timeout we risk to slow down the endpoint chain.

Cannot you modify your PR to do something like this?

import httpx

timeout = httpx.Timeout(timeout=5)
timeout.read = 120.0

client = httpx.AsyncClient(base_url="https://www.bing.de", timeout=timeout)

print(client.timeout.as_dict())
Johann-Foerster commented 7 months ago

Agree! Unfortunately, probably more changes needed https://github.com/timoklimmer/powerproxy-aoai/issues/37

timoklimmer commented 7 months ago

Thanks, Johann. I have adopted your suggestion and have changed the read timeout.