Open techcow2 opened 1 month ago
This is a great idea. I think that for starters a button that says "continue" and automatically shows up when max_tokens is hit, could be a good start. If people like and use it, we can apply a couple of variations: Anthropic actually lets you continue an answer (is called prefill), while for Google and OpenAI it's not as deterministic, but I may be able to make this work.
What do you think: Phase 1. Auto-detection of interruption & surfacing of a "Continue" button Phase 2. Continuation mechanic: do not use a "continue" user message, but use a pre-fill or post-merge llm mechanic Phase 3. If this works well, auto-do it (e.g. loop of detection and prefill until the end) ?
This is a great idea. I think that for starters a button that says "continue" and automatically shows up when max_tokens is hit, could be a good start. If people like and use it, we can apply a couple of variations: Anthropic actually lets you continue an answer (is called prefill), while for Google and OpenAI it's not as deterministic, but I may be able to make this work.
What do you think: Phase 1. Auto-detection of interruption & surfacing of a "Continue" button Phase 2. Continuation mechanic: do not use a "continue" user message, but use a pre-fill or post-merge llm mechanic Phase 3. If this works well, auto-do it (e.g. loop of detection and prefill until the end) ?
Thank you for considering my request. This sounds great! The only other thing I would like to suggest is that you continue the code from where it left off within the same block, as if there were no interruption. This way, copying and pasting the code once it's complete can be done seamlessly, without having to figure out exactly where the code starts and stops due to the interruption.
I've added interruption detection for 10 model vendors (LocalAI and LMStudio do not emit max-tokens as the finish reason, nor emit token counts).
See the "Cont" column in the table below.
No UI support yet, but it's really easy now to show a continuation button.
@techcow2 basic UI control for the continuation. Very useful to understand the reason for stopping, indeed.
https://github.com/user-attachments/assets/a3f8f40d-eea6-4dd6-9093-0e4ae77da3b1
@techcow2 basic UI control for the continuation. Very useful to understand the reason for stopping, indeed.
Recording.2024-08-01.054151.mp4
Wow! Thanks for getting this started so quickly. Is there a way to merge the text generated after clicking 'continue' into the same block where the text initially stopped?
Why Creates a less painful, and disruptive experience for coders. Also, if implemented, I believe this would be the only front end to offer such a feature.
Description When the LLM stops generating code that is extensive, create an option that automatically tells the LLM to 'continue' without the need for user intervention.
Requirements All I can say is to work around this, I currently type 'continue' followed by the latest complete piece of code (to avoid issues with structure. Meaning, I don't want to guess if where it stopped generating is supposed to be the end of the current line)