Continue (Button) or Continue (Audodetect)

[bet0x]

Hello and congrats on such excellent work!

1)

I'm using https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API with the openai API (simil) to openai. To continue generation, I added for testing a small button next to send called "Continue," which sends _continue as a message. Hence, it continues generation, and it works, but dunno how to remove the user message with the _continue from the chat.

2) Another alternative would be reading the generated json and detect if finish_reason eq to length it means the message is not yet complete and just "ask" for the rest. (there is finish_reason eq stop).

Here it is the json to experiment, both versions:

{"id":"chatcmpl-1703102330350430976","object":"chat.completions","created":1703102330,"model":"zephyr-7b-beta.Q6_K.gguf","choices":[{"index":0,"finish_reason":"length","message":{"role":"assistant","content":"Title: Embracing the Power of Linux: The Ultimate Guide\n\nLinux, the open-source operating system, has become a popular choice for IT professionals, developers, and enthusiasts alike. With its flexibility, customizability, and security features, Linux has carved out a niche for itself in the tech landscape. In this article, we’ll explore the benefits of Linux, discuss some popular use cases, and provide a beginner’s guide to get started with this powerful operating system.\n\nBenefits of Linux\n\nLinux offers a host of benefits, many of which make it a top choice for tech-savvy users. Here are some of the major advantages of choosing Linux:\n\n reliability: Linux is renowned for its reliability and stability. With a stable, robust kernel and a modular architecture, Linux ensures that system processes run smoothly and without errors. This makes Linux an ideal choice for mission-critical applications where downtime cannot be tolerated.\n\nsecurity: Linux is a secure operating system with an impressive track record in preventing security breaches. With features like mandatory access control, SELinux, and AppArmor, Linux provides strong protection against attacks. Additionally, the open-source nature of Linux makes it easier to identify and address security issues quickly.\n\ncustomizability: Linux is highly customizable, allowing users to tailor their systems to their specific needs. With a vast range of software packages and tools available, users can easily add new features and functionalities. Moreover, the open-source nature of Linux means that users can contribute to the development of the OS and the software ecosystem, making it a truly community-driven platform.\n\ncost-effectiveness: Linux is a free and open-source operating system, which makes it an attractive choice for individuals and organizations looking to save costs. Apart from the cost-effectiveness, Linux can also reduce hardware requirements since it consumes fewer system resources compared to other OSs.\n\npopular use cases of Linux\n\nLinux has a wide range of applications across various industry sectors. Here are some popular use cases of Linux:\n\nweb servers: Linux is the preferred choice for web servers due to its reliability, scalability, and security features. With Apache, Nginx, and other popular web servers, Linux enables organizations to build and manage high-performance, cost-effective web platforms.\n\ndatabase servers: Linux is an ideal choice for database"}}],"usage":{"prompt_tokens":48,"completion_tokens":513,"total_tokens":561}}

{"id":"chatcmpl-1703102393123772928","object":"chat.completions","created":1703102393,"model":"zephyr-7b-beta.Q6_K.gguf","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"Love is a powerful emotion that connects two people deeply. It can be characterized as a feeling of affection, kindness, and devotion towards someone else. Love can take on many different forms, including romantic love, platonic love, and self-love. It often involves actions and sacrifices that demonstrate compassion, respect, and commitment. Ultimately, love is a fundamental aspect of the human experience that brings people closer together and helps them to thrive as individuals and couples."}}],"usage":{"prompt_tokens":38,"completion_tokens":97,"total_tokens":135}}

Would you mind helping implementing such features?

[iansinnott]

So you're using the oobabooga API as an endpoint in Prompta?

[bet0x]

So you're using the oobabooga API as an endpoint in Prompta?

Yes, it is compatible.

[iansinnott]

Yeah it's meant to be usable with any openai-compatible api. so what's the issue here?

You want to remove user messages from the chat?

If so you can delete those messages via the UI. this was recently shipped. see #14

[bet0x]

Yeah it's meant to be usable with any openai-compatible api. so what's the issue here?

You want to remove user messages from the chat?

If so you can delete those messages via the UI. this was recently shipped. see #14

I wanted to make a PR but i'm not familiar with Svelte, essentially, on the API, there are 2 or maybe more, types of finish_reason:

1. length
2. stop

The first is when you still have content to be received. The second when the message is fully received. I added a simple Continue button to get more content but the message gets on the chat and the idea is to have it as a feature but invisible. The other option is to detect when finish_reason = length is received and then send _continue again.

Example:

https://platform.openai.com/docs/api-reference/chat/streaming const finishReason = parsed.choices[0]?.finish_reason; if finishReason === 'length' then continue.