How to change request timeout for createChatCompletion

[nemoo]

I am using the chat completion function like so: val response: ChatResponse = openAI.createChatCompletion(chatRequestBody)

When I run it against the current top model, 'p1-preview', I run into 30 second timeouts. '01-preview' is designed to take longer to reason, so it is ok if it takes longer.

The timeout I get: java.util.concurrent.TimeoutException: test timed out after 30 seconds

I did not find a parameter on 'createChatCompletion' to increase the timeout. How to do it?

edit: By using a DefaultSyncBackend I can adjust the connection timeout: val backend = DefaultSyncBackend(BackendOptions.connectionTimeout(5.minutes))

But that does not help in my case, because I need a larger request timeout.

[adamw]

The default read timeout is 1 minute, so I'm not sure where the 30 seconds come from. But maybe this will help, it's a per-request setting (as compared to connection timeout):

openAI.createChatCompletion(chatRequestBody).readTimeout(10.minutes)

?

[nemoo]

Thanks!
I am using this 'openAI':

val openAI: OpenAISyncClient = OpenAISyncClient(apiKey)
val response: ChatResponse = openAI.createChatCompletion(chatRequestBody)

It seems to me that OpenAISyncClient.createChatCompletion returns a ChatResponse: https://github.com/softwaremill/sttp-openai/blob/master/core/src/main/scala/sttp/openai/OpenAISyncClient.scala#L185

And function readTimeout is not a member of sttp.openai.requests.completions.chat.ChatRequestResponseData.ChatResponse?

[adamw]

Ah yes, it's not. We should probably add a way to customise the request options to OpenAISyncClient. But let's first debug this case.

Can you try using the OpenAI class directly, to obtain an sttp-request, customise it, and then send it?

val backend = DefaultSyncBackend()
OpenAI(apiKey).createChatCompletion(chatRequestBody).readTimeout(10.minutes).send(backend)

[nemoo]

Interesting. I am now performing the call using your exact code snippet and I still get this timeout: java.util.concurrent.TimeoutException: test timed out after 30 seconds

[nemoo]

but I am using it in a test case. Will try outside of test case

[nemoo]

Aaah yes, that was it. Munit has a default timeout of 30 seconds per test. That was triggered in this case.

So your proposed way works for me for now, thanks! If I can customise the request options in the future directly with OpenAISyncClient it will be even better

Thanks!

[adamw]

See #220