search

Mintplex-Labs / anything-llm

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.
https://anythingllm.com
MIT License
27.45k stars 2.76k forks source link

[FEAT]: Remove the [CONTEXT 0][CONTEXT 2][CONTEXT 5] information #2658

Open morbificagent opened 1 day ago

morbificagent commented 1 day ago

What would you like to see?

Hi together, i am playing around with anythingllm connected to perplexity and its working good, but all responses are containing [CONTEXT 0][CONTEXT 2][CONTEXT 5] inside the text which makes it hard to read.

I have tried to filter it out by prompt but it seems to be impossible.

Would great if anyone has an idea how to filter it out in the response as it doesnt help anyone as nobody knows whats [CONTEXT 5]

lewismacnow commented 1 day ago

What would you like to see?

responses are containing [CONTEXT 0][CONTEXT 2][CONTEXT 5] inside the text which makes it hard to read.

I have tried to filter it out by prompt but it seems to be impossible. [CONTEXT 5]

You can usually work around this with a tweaked System Prompt:

"Given the following conversation, relevant context, and a follow up question, reply with an answer to the current question the user is asking. Return only your response to the question given the above information following the users instructions as needed and do not mention or refer to the "context" specifically in your response. You respond in plain English only as if the provided context is knowledge you are aware of and do not respond with JSON or tool calls, respond like a human speaking to another human."

Another good method is to create a "new" model, with some existing conversation already baked into the model file: Example from Ollama

MESSAGE user Context: [CONTEXT 0]: Grass is green [END CONTEXT 0] [CONTEXT 1]: The Sky is blue [END CONTEXT 1] Is the Sky green? MESSAGE assistant Last time I checked, the sky was blue. Grass, on the other hand, is green!

timothycarambat commented 1 day ago

i am playing around with anythingllm connected to perplexity and its working good

How are you using AnythingLLM? The context is in the system prompt so that shouldn't be showing in an LLM response, and if it does that is just a model behavior. I haven't had it return context snippets in the response before

morbificagent commented 22 hours ago

Hi together and thanks for the answers,

i have read a little bit and it looks like perplexity is delivering these [context x] "informations" inside the streamed text response. And saying "dont deliver this..." in the prompt doesnt help.

Here are the tokens that are coming from perplexity:

[backend] info: Original Token: -Wissen [backend] info: Original Token: darst [backend] info: Original Token: ellt.[CONTEXT [backend] info: Original Token: 1][CONTEXT [backend] info: Original Token: 31][CONTEXT [backend] info: Original Token: 54]

So i think there isnt a good way to filter them out.

timothycarambat commented 11 hours ago

Ah, seems like a model behavior. I wonder if swapping that to XML would help. Lots of training data is XML/HTML so it may be better are stripping that out of the response. What perplexity model are you using so i can try to repro?