search

enricoros / big-AGI

Generative AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. It features AI personas, AGI functions, multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.
https://big-agi.com
MIT License
5.22k stars 1.18k forks source link

[Roadmap] Continuation feature that benefits coders #613

Open techcow2 opened 1 month ago

techcow2 commented 1 month ago

Why Creates a less painful, and disruptive experience for coders. Also, if implemented, I believe this would be the only front end to offer such a feature.

Description When the LLM stops generating code that is extensive, create an option that automatically tells the LLM to 'continue' without the need for user intervention.

Requirements All I can say is to work around this, I currently type 'continue' followed by the latest complete piece of code (to avoid issues with structure. Meaning, I don't want to guess if where it stopped generating is supposed to be the end of the current line)

enricoros commented 1 month ago

This is a great idea. I think that for starters a button that says "continue" and automatically shows up when max_tokens is hit, could be a good start. If people like and use it, we can apply a couple of variations: Anthropic actually lets you continue an answer (is called prefill), while for Google and OpenAI it's not as deterministic, but I may be able to make this work.

What do you think: Phase 1. Auto-detection of interruption & surfacing of a "Continue" button Phase 2. Continuation mechanic: do not use a "continue" user message, but use a pre-fill or post-merge llm mechanic Phase 3. If this works well, auto-do it (e.g. loop of detection and prefill until the end) ?

techcow2 commented 1 month ago

This is a great idea. I think that for starters a button that says "continue" and automatically shows up when max_tokens is hit, could be a good start. If people like and use it, we can apply a couple of variations: Anthropic actually lets you continue an answer (is called prefill), while for Google and OpenAI it's not as deterministic, but I may be able to make this work.

What do you think: Phase 1. Auto-detection of interruption & surfacing of a "Continue" button Phase 2. Continuation mechanic: do not use a "continue" user message, but use a pre-fill or post-merge llm mechanic Phase 3. If this works well, auto-do it (e.g. loop of detection and prefill until the end) ?

Thank you for considering my request. This sounds great! The only other thing I would like to suggest is that you continue the code from where it left off within the same block, as if there were no interruption. This way, copying and pasting the code once it's complete can be done seamlessly, without having to figure out exactly where the code starts and stops due to the interruption.

enricoros commented 1 month ago

I've added interruption detection for 10 model vendors (LocalAI and LMStudio do not emit max-tokens as the finish reason, nor emit token counts).

See the "Cont" column in the table below. image

No UI support yet, but it's really easy now to show a continuation button.

enricoros commented 1 month ago

@techcow2 basic UI control for the continuation. Very useful to understand the reason for stopping, indeed.

https://github.com/user-attachments/assets/a3f8f40d-eea6-4dd6-9093-0e4ae77da3b1

techcow2 commented 1 month ago

@techcow2 basic UI control for the continuation. Very useful to understand the reason for stopping, indeed.

Recording.2024-08-01.054151.mp4

Wow! Thanks for getting this started so quickly. Is there a way to merge the text generated after clicking 'continue' into the same block where the text initially stopped?