[BUG]: Getting mostly empty answers when using Generic OpenAI

[HuidaQ]

How are you running AnythingLLM?

AnythingLLM desktop app

What happened?

When using Generic OpenAI option in LLM Preference, most answers I got was either empty or super short/incomplete. See screenshots below.

Incomplete answers: (the Lora and QLora papers are added to the workspace)

LLM setting:

Are there known steps to reproduce?

See screenshots above.

[shatfield4]

What LLM API are you using here? This tends to happen when the LLM API you are connecting to does not form the streaming chunks the same way as OpenAI does. If we can get access to that API you are using, we can create a custom integration for that specific API so it can handle the streaming chunks differently and not end the stream after just a few tokens.

[HuidaQ]

I'm using the lingyiwanwu's API. See https://platform.lingyiwanwu.com/docs (sorry the page is in Chinese). I can't expose my own key but they have free trials (up to 36 RMB) once signed up.

API call example from their docs:

import openai
from openai import OpenAI
API_BASE = "https://api.lingyiwanwu.com/v1"
API_KEY = "your key"
client = OpenAI(
    api_key=API_KEY,
    base_url=API_BASE
)
completion = client.chat.completions.create(
    model="yi-large",
    messages=[{"role": "user", "content": "Hi, who are you?"}]
)
print(completion)

[timothycarambat]

This will be fixed in the next version of the desktop app. This was already patched with PR #1487 :+1