danneu / telegram-chatgpt-bot

a Telegram ChatGPT bot that supports text prompts and two-way voice memos
33 stars 8 forks source link

Add a toggle for message streaming vs buffering #7

Closed danneu closed 1 year ago

danneu commented 1 year ago

The bot used to use the OpenAI chat completion API where the response would be received once the entire ChatGPT answer was buffered on OpenAI's end.

Recently I changed this to use stream=true and repeatedly update the Telegram message as answer chunks are received.

This is really nice on desktop. But on mobile, due to how Telegram handles a message that's repeatedly edited to grow longer and longer, it can be hard to read for long answers.

Perhaps the toggle command can be /stream.

danneu commented 1 year ago

This is easy to implement:

    let result: { answer: string; messageId: number; tokenCount: number }
    if (chat.stream) {
        result = await streamTokensToTelegram(
            chatId,
            messageId,
            tokens,
            chat.model,
        )
    } else {
        let tokenCount = 0
        let answer = ''
        for await (const token of tokens) {
            tokenCount++
            answer += token
        }
        const message = await telegram.sendMessage(chatId, answer, messageId)
        result = {
            answer,
            messageId: message.message_id,
            tokenCount,
        }
    }

But waiting for the full response to buffer is such bad UX I think I'd rather always stream.