We want to investigate using litellm with prompt_toolkit. We'd like to stream data that is coming in via the stream option in litellm to update the text in the terminal as it arrives, as opposed to doing it in chunks (where webwright waits until it has the whole thing to display).
As we use a formatter for the text that we get back from the LLM (after it is done) we also need to investigate if we can switch the markup/style on the text as it streams out into the console.
Implement a separate reference application to investigate this. Start by getting a simple terminal prompt running in prompt_toolkit, then wire up litellm to talk to an LLM (probably OpenAI) and then see if you can stream the result into the console when it replies. When it is done, it should return you to the prompt to continue the conversation.
You might also want to investigate capturing the ctrl-c keystroke, and instead of exiting the app, interupt the stream and dump the user back to the prompt. To exit the app, we could type exit (or hit ctrl-c from the active prompt). Aider uses two control cs in a row for this, but we can do something else.
kordless
commented
1 month ago
https://github.com/BerriAI/litellm
We want to investigate using litellm with prompt_toolkit. We'd like to stream data that is coming in via the stream option in litellm to update the text in the terminal as it arrives, as opposed to doing it in chunks (where webwright waits until it has the whole thing to display).
As we use a formatter for the text that we get back from the LLM (after it is done) we also need to investigate if we can switch the markup/style on the text as it streams out into the console.
Implement a separate reference application to investigate this. Start by getting a simple terminal prompt running in prompt_toolkit, then wire up litellm to talk to an LLM (probably OpenAI) and then see if you can stream the result into the console when it replies. When it is done, it should return you to the prompt to continue the conversation.
You might also want to investigate capturing the ctrl-c keystroke, and instead of exiting the app, interupt the stream and dump the user back to the prompt. To exit the app, we could type exit (or hit ctrl-c from the active prompt). Aider uses two control cs in a row for this, but we can do something else.