Stream error after 5 minutes

[EthanBlackburn]

Describe the bug

Streaming output from gpt4 consistently fails with Connection broken: InvalidChunkLength(got length b'', 0 bytes read) for prompts that trigger long-form responses. I've been able to reproduce this by prompting gpt to rewrite a long document.

A few observations 1) This failure occurs exactly at the 5 minute mark every time, which leads me to believe there's a timeout somewhere. 2) The python implementation of stream reader assumes that an empty string will never be returned by the server. 3) I sent the same request using a non-streaming client with a timeout at 10min. The timeout wasn't triggered until the 10min mark,so this problem seems isolated to streaming

Code snippet from urllib3/response.py:

def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()  # self._fp.fp.readline() returns b''
        line = line.split(b";", 1)[0]
        try:
            self.chunk_left = int(line, 16)  # this raises a value error
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise InvalidChunkLength(self, line)

It seems like a problem on the server side but Im not confident

To Reproduce

1. Create gpt4 streaming client
2. Prompt it to rewrite a very long text. For example, verses 1:1 -> 4:10 in the book of genesis

Code snippets

No response

OS

macOs

Python version

Python 3.10.9

Library version

v.0.26.4

[karim-attia]

I have the exact same issue. I can generate a little over 2'000 tokens via GPT-4 stream and I stop receiving stream updates after exactly 300 seconds.

Running on something similar as this code: https://github.com/Nutlope/twitterbio/blob/e799871764e559cd96608bfefcc3976e7daf346a/utils/OpenAIStream.ts#L26

[oomti]

The default timeout of the connection with aiohttp library(that openai librayr utilizes) is 300 seconds.

Have you changed the timeout period when you send your request?

It's an optional parameter timeout_period and you set it's value in seconds.

[oomti]

@EthanBlackburn if you provide me a generator prompt/model-name for a long text I can test in my custom API client if the connection is closed from server side or it's an issue with the openAPI python library.

[karim-attia]

@oomti This prompt should exceed 5min with GPT-4.

Please write a JSON that counts from 1 to 500 like this: { "One": 1, "Two": 2, }

[oomti]

Thanks @karim-attia

Tested with a 600seconds timeout on my part.

After 5 minutes the stream has ended with the following error message:

openai-chatgpt\openai\Lib\site-packages\aiohttp\streams.py", line 304, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

This indicates that the socket from the serverside is disconnected after 5 minutes, which is probably a default setting.

[oomti]

Doublechecked and tested with various settings - timeout=600, stream=true - connection closed in 5 minutes timeout=600, stream=false - connection closed in 5 minutes(so won't receive a response)

Also tested with the official web UI - completes only 87 lines and closes response after(there might be other limitations there)

I think the large token count was designed with a larger input in mind and a large sized output was not considered.

[yannoth]

Same bug here

[ficapy]

same here

requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

________________________________________________________
Executed in  301.97 secs    fish           external
   usr time    2.23 secs    0.09 millis    2.23 secs
   sys time    1.41 secs    1.76 millis    1.40 secs

[KaedinLian]

I met the same issue too with 4000 toknes prompt and asking near 4000 toknes. timeout=1000, stream=true - connection closed in 5 minutes (call: invalid chunk Connection broken: InvalidChunkLength(got length b'', 0 bytes read) timeout=1000, stream=false - connection closed in 5 minutes(so won't receive a response)

[jizs]

We find out this is not a bug of client, but a bug of the openai server

[wdeng]

I have the exact same error, and interact with ChatGPT and documentation, it seems to be a server side error.

" From the information you've provided, it does indeed seem that there's a timeout issue with the OpenAI GPT-4 streaming API that's closing the connection after exactly 5 minutes (300 seconds). This could be due to a server-side setting.

As for the error you're encountering, InvalidChunkLength(got length b'', 0 bytes read), it's likely that when the server-side socket disconnects after 5 minutes, the client is still trying to read from the connection and encountering an empty string, which leads to this error. This seems to be supported by the fact that the error is thrown when trying to parse an empty string as an integer, which is invalid.

However, without having access to the underlying implementation of the OpenAI API, it's hard to know for sure. It's also worth noting that as an AI developed by OpenAI, I don't have the ability to access or manipulate the OpenAI servers in real-time.

As I recommended earlier, the best course of action would be to reach out to OpenAI's support or raise an issue on the OpenAI GitHub repository. They would be best equipped to address this issue and potentially provide a fix or workaround.

""

[wdeng]

anyone has possible solutions?

[takatost]

I have the same issue when I generate long text using GPT-4.

[JackWH]

Same issue with long GPT-4 outputs, although I'm using the PHP client. After extensive debugging I'm pretty certain the issue is server-side with OpenAI, but besides this thread, this is the only place I've found it being discussed.

The issue seems to affect both standard and streamed responses. With standard responses on nginx, the OpenAI server closes the connection after 5 minutes, and nginx reports this as a Gateway Timeout.

With streamed responses, the final delta doesn't specify a finish_reason, suggesting the model is still running. But the stream is terminated anyway.

I can reproduce this on both macOS and Ubuntu. Changing nginx directives makes no difference, nor does overriding HTTP client timeouts.

This appears to be a recent change, it worked as expected for me up until last week. Hopefully this is fixed soon — I've sent a link to this comment to OpenAI's support, hopefully someone sees it 🤞

[gabrilator]

hey! any news regarding this? having the same problem with long text and gpt 4!!

[t4ng]

I encountered the same issue of automatic interruption after 300 seconds. Is this a bug or a feature?

[JackWH]

@t4ng Until OpenAI acknowledge or document it somewhere, it's definitely a bug — as it prevents many use cases for longer 8K/32K context windows. It worked fine in the past, but appears to be a recent fault.

I submitted a support ticket but have yet to receive a response. If anyone else is able to do so too, this may get picked up sooner. It seems we're still being charged for the full generation, even though we only receive a truncated version.

[dyar-al-ashtari]

I have the same issue. It disconnects after 5 minutes exactly. A temporary fix is to ask GPT-4 only to send you half of its response (and hoping it takes less than 5 minutes) and send you the remaining half when you prompt it "NEXT."

[cantrell]

I'll add my voice to the chorus . I've been struggling with this error for a couple of weeks now. I've come up with some workarounds (asking the model to do less so that it completes in less time), and I'm even working on a code-based workaround, but I'm hoping that this gets fixed soon since any form of workaround comes at a cost both literally (in terms of API usage) and user experience.

(For those just discovering this thread, it's worth noting that this is not a bug with the OpenAI library. I'm using the REST APIs directly; the bug is with OpenAI's servers. Also, at least in the JavaScript world, the resulting network error cannot be caught and handled gracefully.)

[treya-lin]

Does anyone have any update regarding this issue? I met this error when using gpt-4-0613 and gpt-3.5-turbo-16k-0613 with stream=True :(

[treya-lin]

I read this post too https://community.openai.com/t/how-to-set-a-timeout-on-an-api-function-call-using-using-the-python-library/4213/4, but I'm confused at the parameter's name, should it be timeout or request_timeout? Or they both exist and function differently?

By the way, timeout-relevant parameters were not mention in openai's api documentation https://platform.openai.com/docs/api-reference/chat/create, so I didn't realize these parameters were available in the first place.... I have to say I am so overwhelmed with all the scattered documentations... what api documentations do you guys refer to mostly ...

[JackWH]

@treya-lin

By the way, timeout-relevant parameters were not mention in openai's api documentation https://platform.openai.com/docs/api-reference/chat/create

The timeout parameters you mentioned aren’t to do with the API, they’re specific to the Python client (hence not mentioned in the API docs). Those timeout params just provide a way to close the connection after a fixed time.

The 5-minute timeout issue discussed here is unrelated to any API parameters, and seemingly beyond our control, apparently due to an OpenAI server misconfiguration.

[Oliver-ss]

mark

[jzhutwilio]

Meet the same issue, any help/workaround will be appreciated!

[ShantanuNair]

Facing this issue as well.

[ShantanuNair]

@oomti , I see the same error you do. Can you confirm the OP issue is related? Did you find a fix?

[rattrayalex]

This is fixed in v1; please upgrade to the latest version of the library.

[cantrell]

This is fixed in v1; please upgrade to the latest version of the library.

That's great news! @rattrayalex: Can you share why upgrading to the newest version of the library fixes the issue? The problem is (was?) with the underlying REST API, so either the servers' configuration has changed, or REST users need to update their code. (I suspect the former.)

[rattrayalex]

I believe there was an issue in the client (no longer there) and a limitation on the server side which has also been improved since then (other users have reported they hit a timeout at 10min recently).