Streaming support not compatible in v2

[XInTheDark]

Hi! So my code looks like this:

export const getGoogleGeminiResponse = async (chat, options, stream_update, max_retries = 5) => {
  const APIKey = getPreferenceValues()["GeminiAPIKey"];
  const googleGemini = new Gemini(APIKey, { fetch: fetch });
  let formattedChat = GeminiFormatChat(chat);

  // Create chat
  let query = chat[chat.length - 1].content;
  const geminiChat = googleGemini.createChat({
    model: options.model,
    messages: formattedChat,
    maxOutputTokens: 8192, // accurate as of 2024-05-31, for gemini-1.5-flash-latest
    temperature: options.temperature || 0.7,
  });

  // Send message
  try {
    let response = "";
    if (stream_update) {
      const handler = (chunk) => {
        response += chunk;
        stream_update(response);
      };
      await geminiChat.ask(query, { stream: handler });
    } else {
      response = await geminiChat.ask(query);
      return response;
    }
  } catch (e) {
    if (max_retries > 0) {
      console.log(e, "Retrying...");
      return await getGoogleGeminiResponse(chat, options, stream_update, max_retries - 1);
    } else {
      throw e;
    }
  }
};

Pretty standard stuff, and it works perfectly with v1.1.0 of your API. But with the exact same code, after upgrading to v2.0.0 and later, I get the following error:

 Error: Error: TypeError: obj[__knownSymbol(...)] is not a function
    at __forAwait (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:28906:136)
    at handleReader (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:28948:23)
    at _Gemini2.handleStream (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:29118:13)
    at _Gemini2.ask (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:29248:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Chat.ask (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:29304:24)
    at async getGoogleGeminiResponse (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:29347:7)
    at async chatCompletion (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:29423:16)
    at async getChatResponse (/Users/jerry/.config/raycast/extensions/raycast-g4f/aiChat.js:29451:10)
    at async updateChatResponse (/Users/jerry/.config/raycast/extensions/raycas<…>

As the new API is supposed to be fully compatible with previous versions, could you check if something has been changed? I'm also happy to provide more debug details if you need them. Thanks in advance!

[EvanZhouDev]

This potentially appears to be a fetch polyfill issue. I realize that you are using Raycast and need to polyfill fetch (I develop extensions for it too). I recommend using the node-fetch polyfill, as others may not work too well. The entirety of streaming was redesigned (not to mention rewritten) with SSE, so it makes sense that there may be minor differences. Please attempt to use node-fetch, and if you are/it still doesn't work, please let me know!

I will try to see if I can make fallback streaming implementations for different polyfills later.

Sorry for the inconvenience about the incompatibility, by the way... it's rather difficult to keep it completely backwards compatible especially through a change in implementation and a rewrite.

[XInTheDark]

Thanks for the solution! I changed to node-fetch and everything works great now :)

[EvanZhouDev]

Do you mind sharing which polyfill you used before so I can debug further?

[XInTheDark]

Sure! I used node-fetch-polyfill, latest version. The only difference was which fetch function was passed when initialising the Gemini object.

[EvanZhouDev]

Ah yes, I can reproduce now (I was using my own API wrong 🤦‍♂️). I am implementing a fallback solution (which works for native fetch and node-fetch-polyfill but not node-fetch) in case AsyncIterators are not defined (i.e. for node-fetch-polyfill or Safari native fetch) so that it should be a complete streaming support for all environments.

[EvanZhouDev]

@XInTheDark The fix is now applied, with a direct response.body.getReader() used as the default (which works for most fetch polyfills except node-fetch), then if that fails, the fallback is AsyncIterator, which works with a lot less environments, but it does for node-fetch. It is available in Gemini AI v2.1.1