Chatgpt api (openai-node v4.26.0) stream issue with gpt-4 models

[ElicaInc]

Hi! 👋

Firstly, thanks for your work on this project! 🙂

Today I used patch-package to patch react-native-sse@1.2.0 for the project I'm working on.

Some words and characters are experiencing duplication in ChatGPT API + react-native-sse.

I was able to fix it in my setting by rewriting as follows. However, it has not been confirmed whether it works in other environments.

Here is the diff that solved my problem:

diff --git a/node_modules/react-native-sse/src/EventSource.js b/node_modules/react-native-sse/src/EventSource.js
index ee03edc..27ce2bd 100644
--- a/node_modules/react-native-sse/src/EventSource.js
+++ b/node_modules/react-native-sse/src/EventSource.js
@@ -33,6 +33,7 @@ class EventSource {
     this.body = options.body || undefined;
     this.debug = options.debug || false;
     this.interval = options.pollingInterval ?? 5000;
+    this.preResponseLength = null; // added

     this._xhr = null;
     this._pollTimer = null;
@@ -174,7 +175,13 @@ class EventSource {
   }

   _handleEvent(response) {
-    const parts = response.substr(this.lastIndexProcessed).split('\n');
+    const parts = response.substr(this.preResponseLength ?? 0).split('\n');
+    this.preResponseLength = response.length;

     const indexOfDoubleNewline = response.lastIndexOf('\n\n');
     if (indexOfDoubleNewline != -1) {

This issue body was partially generated by patch-package.

[EmilJunker]

Hi, thank you for raising attention for this interesting issue. Could you please check if the changes made in this open pull request fix the problem: #39. It looks to me like they might.

[ElicaInc]

Thank you for your suggestion. However, the proposed changes did not result in the correction.

    const indexOfDoubleNewline = response.lastIndexOf('\n\n');
     if (indexOfDoubleNewline < 0) {
       return;
     }
     this.lastIndexProcessed = indexOfDoubleNewline + 2;

[ElicaInc]

The potential error may have occurred within the substr( this.lastIndexProcessed) operation. Here is the logs of parts:

 LOG  ["data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"logprobs\":null,\"finish_reason\":null}]}", ""]
 LOG  ["data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"role\":\"assistant\",\"content\":\"\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"Hello\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}]}", ""]
 LOG  ["data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" How\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" can\"},\"logprobs\":null,\"finish_reason\":null}]}", ""]
 LOG  ["data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" can\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" I\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" assist\"},\"logprobs\":null,\"finish_reason\":null}]}", ""]
 LOG  ["data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" assist\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" you\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\" today\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"?\"},\"logprobs\":null,\"finish_reason\":null}]}", "", "data: {\"id\":\"chatcmpl-8sj...Ch1\",\"object\":\"chat.completion.chunk\",\"created\":1708054070,\"model\":\"gpt-4-0125-preview\",\"system_fingerprint\":\"fp_5...6\",\"choices\":[{\"index\":0,\"delta\":{},\"logprobs\":null,\"finish_reason\":\"stop\"}]}", "", "data: [DONE]", "", ""]

[EmilJunker]

@ElicaInc Thank you for the response and for providing these logs. I think I understand now what causes this problem. I really think that the changes made in #39 will fix this. Please try it again with all the changes from here: https://github.com/binaryminds/react-native-sse/pull/39/files

[ElicaInc]

Hi @EmilJunker, Unfortunately, #39 did not work well, although I tried again.

const parts = response.substr(this.lastIndexProcessed).split('\n');

const indexOfDoubleNewline = response.lastIndexOf('\n\n');
if (indexOfDoubleNewline <= (this.lastIndexProcessed - 2)) {
   return;
}

this.lastIndexProcessed = indexOfDoubleNewline + 2;

[EmilJunker]

Sorry, my mistake. Even though the changes proposed in #39 are meant to fix this exact problem, they are not actually sufficient when several event data blocks are added simultaneously and the last one is missing the extra newline.

To fix this problem for good (in a way that doesn't cause problems in other environments), please replace the beginning part of the _handleEvent function with this:

_handleEvent(response) {
  const indexOfDoubleNewline = this._getLastDoubleNewlineIndex(response);

  if (indexOfDoubleNewline <= this.lastIndexProcessed) {
    return;
  }

  const parts = response.substring(this.lastIndexProcessed, indexOfDoubleNewline).split('\n');
  this.lastIndexProcessed = indexOfDoubleNewline;

And add this new function:

_getLastDoubleNewlineIndex(text) {
  const lastNN = text.lastIndexOf('\n\n');
  if (lastNN === -1) {
    return -1;
  }
  return lastNN + 2;
}

[ElicaInc]

It works.

[wojciechkrol]

Hello everyone. Thank you for pointing this out! @EmilJunker could you create a PR with your solution?

[EmilJunker]

@EmilJunker could you create a PR with your solution?

I will.