Issue with GPT-4 Model When Processing Large Laravel Application Prompt

[gileneusz]

I encountered an issue while using the GPT-4 model for processing a quite extensive and intricate prompt related to a Laravel application setup.

Issue:

When I used the model to generate output based on the following prompt:

We are writing an application in Laravel for the backend, inertia, and Vue for the frontend. This application is for testing doctors' final exam with ABCDE single answer tests. Those tests will be split into categories - such as specialties. All files such as models, controllers, migrations, views etc. are split into separate files or folders according to Laravel folders structure.

The application returned an error after a few minutes of processing, specifically the requests.exceptions.ChunkedEncodingError was thrown. This suggests an issue related to the handling of chunked encoding in the HTTP response.

Based on the traceback, the error seems to originate from urllib3, where it's trying to read a chunk of the HTTP response, but encounters an invalid chunk length. This error may be a potential indication that the application has reached the context window limit of 8192 tokens set for the GPT-4 model.

Another possible cause could be related to network limitations or rate restrictions associated with GPT-4, which might be causing the issue when handling larger inputs or running longer processes.

Reproduction:

The issue can be reproduced by running the aforementioned prompt in your application with the GPT-4 model.

Please investigate this issue, as it is causing some difficulties in handling larger prompts and processing them for extended periods in the application. If more information is required, feel free to ask.

Thank you for your attention to this matter.

Here's error log:

Traceback (most recent call last):

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 761, in _update_chunk_length
    self.chunk_left = int(line, 16)

ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 444, in _error_catcher
    yield

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 828, in read_chunked
    self._update_chunk_length()

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 765, in _update_chunk_length
    raise InvalidChunkLength(self, line)

urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/user/anaconda3/lib/python3.10/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 624, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 816, in read_chunked
    with self._error_catcher():

  File "/Users/user/anaconda3/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)

  File "/Users/user/anaconda3/lib/python3.10/site-packages/urllib3/response.py", line 461, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)

urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/user/anaconda3/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,

  File "/Users/user/anaconda3/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)

  File "/Users/user/www/engineer/gpt-engineer/gpt_engineer/main.py", line 63, in <module>
    app()

  File "/Users/user/www/engineer/gpt-engineer/gpt_engineer/main.py", line 58, in main
    messages = step(ai, dbs)

  File "/Users/user/www/engineer/gpt-engineer/gpt_engineer/steps.py", line 16, in simple_gen
    messages = ai.start(

  File "/Users/user/www/engineer/gpt-engineer/gpt_engineer/ai.py", line 29, in start
    return self.next(messages)

  File "/Users/user/www/engineer/gpt-engineer/gpt_engineer/ai.py", line 53, in next
    for chunk in response:

  File "/Users/user/anaconda3/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 166, in <genexpr>
    return (

  File "/Users/user/anaconda3/lib/python3.10/site-packages/openai/api_requestor.py", line 692, in <genexpr>
    return (

  File "/Users/user/anaconda3/lib/python3.10/site-packages/openai/api_requestor.py", line 115, in parse_stream
    for line in rbody:

  File "/Users/user/anaconda3/lib/python3.10/site-packages/requests/models.py", line 865, in iter_lines
    for chunk in self.iter_content(

  File "/Users/user/anaconda3/lib/python3.10/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)

requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

[patillacode]

AFAIU we are limited by Open AI's limits.

I would suggest you use the 16k model gpt-4-0613 I think it is, but you might find the same issue if your application is too big.

[gileneusz]

Thank you for your prompt response and suggestion.

I understand that we are working within the constraints set by OpenAI. However, at this moment, I don't have access to the 16k model of GPT-4. I attempted to run the process using the gpt-3.5-16k model, and unfortunately, the same error occurred.

Given this context, I was wondering if introducing some form of token limitation per minute in the application might provide a solution. This could potentially keep us within the context window limit and prevent the chunked encoding error we're experiencing.

[krrishdholakia]

Hey @gileneusz,

Happy to add this to reliableGPT (wrapper for the openai endpoint). How would you envision the token limitation working?

Source: https://github.com/BerriAI/reliableGPT