[Bug]: chatgpt答复被频繁截断，随着服务端运行时间增加情况越来越严重

[lrx0014]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

最开始的时候一切正常，程序能够正常答复一段很长的回答，然后过了大概3天开始出现文字截断的现象，需要通过不停输入"继续"来获取剩下的文字。上图是程序运行第7天，情况已经发展到每隔几个字就被截断，几乎无法使用了。

目前发现重启程序重新连接openai的api服务就可以完全恢复正常，在 #393 虽然提到了可能是openai自身设了限制，但我觉得每隔几个字就被截断并且只能通过重启解决有点点诡异。不知道有没有除了定时重启之外的解决方法

p.s 该openai账号是我自己的，目前只有我一个人用，非共享账号，平均每天使用不超过10次，没有大文本对话，基本上一个提问20个字左右

Steps to reproduce the problem

1. 使用revChatGPT.V3实现了一个slack webhook对接到slack机器人
2. webhook程序在运行一段时间后开始出现截断现象，即使是短文本也很严重，重启程序可以解决
3. ...

What should have happened?

短文本的问答场景不应该出现这么频繁的截断现象

Version where the problem happens

Name: revChatGPT Version: 3.3.4 Summary: ChatGPT is a reverse engineering of OpenAI's ChatGPT API Home-page: https://github.com/acheong08/ChatGPT Author: Antonio Cheong Author-email: acheong@student.dalat.org License: GNU General Public License v2.0 Location: /usr/local/lib/python3.11/site-packages Requires: httpx, OpenAIAuth, prompt-toolkit, requests, tiktoken

What Python version are you running this with?

Python 3.11.2

What is your operating system ?

No response

Command Line Arguments

no

Console logs

----------------------------------------
Exception occurred during processing of request from ('*.*.*.*', 48862)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.11/http/server.py", line 667, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__
    self.handle()
  File "/usr/local/lib/python3.11/http/server.py", line 432, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.11/http/server.py", line 400, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('*.*.*.*', 47363)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.11/http/server.py", line 667, in __init__
    super().__init__(*args, **kwargs)
  File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__
    self.handle()
  File "/usr/local/lib/python3.11/http/server.py", line 432, in handle
    self.handle_one_request()
  File "/usr/local/lib/python3.11/http/server.py", line 400, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------
----------------------------------------

Additional information

No response

[Pengchengistaken]

same issue.

[Zero6992]

I previously brought up the same question in #1172, but I haven't noticed anyone else asking about it. Maybe it's only relevant to Mandarin?

[lean-wang]

I have the same problem

[chrisguoado]

You should be aware that there's a token limit for the official api, e.g. 4k token for gpt-3.5-turbo, that's approximate 16k English characters, or 8k Chinese characters, which is the whole context size you can get. Without explicitly resetting the conversation/context, you send the entire conversation (all history prompts and replies) to the openai api each time when you are making a request. If your context has already reached or very close to the limit, then the room left for the response is quite limited. That's why you got truncated replies.

[lrx0014]

https://community.openai.com/t/chat-gpt-3-producing-incomplete-summaries-and-stopping-early-in-text-processing/24490/3

在这个帖子里找到了关于max_tokens的规则，似乎当tokens被耗尽或者不够的时候会导致截断。因此我有个猜测，是不是我们所有的对话都是共用了同一个tokens的额度，tokens耗尽后开始出现截断现象。这也大致可以解释为什么是运行一段时间过后才出现问题，并且重启就好了（可能重启会重置max_tokens？）

def __init__(
        self,
        api_key: str,
        engine: str = os.environ.get("GPT_ENGINE") or "gpt-3.5-turbo",
        proxy: str = None,
        max_tokens: int = 3000, #这里默认3k tokens
        temperature: float = 0.5,
        top_p: float = 1.0,
        presence_penalty: float = 0.0,
        frequency_penalty: float = 0.0,
        reply_count: int = 1,
        system_prompt: str = "You are ChatGPT, a large language model trained by OpenAI. Respond conversationally",
    )

[lean-wang]

你应该知道官方 api 有一个令牌限制，例如 4k 令牌gpt-3.5-turbo，大约 16k 个英文字符，或 8k 个中文字符，这是你可以获得的整个上下文大小。在没有明确重置对话/上下文的情况下，每次发出请求时，您都会将整个对话（所有历史提示和回复）发送到 openai api。如果你的上下文已经达到或非常接近极限，那么留给响应的空间就相当有限了。这就是为什么你得到截断的答复。

So how can I reset the context dialog?

[chrisguoado]

call this:

    def reset(self, convo_id: str = "default", system_prompt: str = None) -> None:
        """
        Reset the conversation
        """
        self.conversation[convo_id] = [
            {"role": "system", "content": system_prompt or self.system_prompt},
        ]

[lean-wang]

reset

thankyou

[manesec]

Do you want to add auto reset conversation function?

[acheong08]

I previously brought up the same question in #1172, but I haven't noticed anyone else asking about it. Maybe it's only relevant to Mandarin?

Seems so. It's probably a token counting issue

[acheong08]

It's meant to delete old messages once they go over token limit but Chinese might not be encoded correctly. A solution is to lower the max_tokens

[olegsvs]

Max_tokens(1500) did not solve the problem =(

[manesec]

same as me, max_tokens 1500 did not solve the problem.

[acheong08]

Might not be tokens then.

ConnectionResetError: [Errno 104] Connection reset by peer

This kinda implies that it's an issue from OpenAI.

[acheong08]

You guys can try https://pypi.org/project/openai/ and see if the same error occurs with long conversations

[chrisguoado]

Max_tokens(1500) did not solve the problem =(

Yeah, simply lower the max tokens cannot solve the problem, once the context has reached the limit, it'll always be kept at the edge of the limit, and you get very little room for the reply, because of this in the request payload:

"max_tokens": self.get_max_tokens(convo_id=convo_id),

For example, you set max tokens to 1500, after the latest question, total token count has reached 1600, then after calling __truncate_conversation() to drop the very first message in your conversation, token count might come to 1495, thus the return value of self.get_max_tokens() will be 5 (the max token count in the following reply).

[chrisguoado]

The solution is to also reserve some room (by default it's set to 300 tokens) for the reply when doing __truncate_conversation(), see #1206 .