⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.
Right now, if a stop word is reached, because StoppingStream is a wrap over the completion stream, the stop word itself is still added to the context and to the SessionId. This causes the following prompt for the same session to need a whole new session to be created, because the original one with the stopping word does not match, and may cause issues with inference due to the extra token that should be getting ignored.
Right now, if a stop word is reached, because
StoppingStreamis a wrap over the completion stream, the stop word itself is still added to the context and to theSessionId. This causes the following prompt for the same session to need a whole new session to be created, because the original one with the stopping word does not match, and may cause issues with inference due to the extra token that should be getting ignored.