Closed danneu closed 1 year ago
This is easy to implement:
let result: { answer: string; messageId: number; tokenCount: number }
if (chat.stream) {
result = await streamTokensToTelegram(
chatId,
messageId,
tokens,
chat.model,
)
} else {
let tokenCount = 0
let answer = ''
for await (const token of tokens) {
tokenCount++
answer += token
}
const message = await telegram.sendMessage(chatId, answer, messageId)
result = {
answer,
messageId: message.message_id,
tokenCount,
}
}
But waiting for the full response to buffer is such bad UX I think I'd rather always stream.
The bot used to use the OpenAI chat completion API where the response would be received once the entire ChatGPT answer was buffered on OpenAI's end.
Recently I changed this to use
stream=true
and repeatedly update the Telegram message as answer chunks are received.This is really nice on desktop. But on mobile, due to how Telegram handles a message that's repeatedly edited to grow longer and longer, it can be hard to read for long answers.
Perhaps the toggle command can be
/stream
.