Streaming custom chat LLM Chain via Flask server

[calypset]

Hi!

First, I would like to say Deep Chat is a great platform for creating versatilie chatbots! Also there is a lot of documentation and examples.

I am trying to stream an LLM Chain with the custom Flask service with chat_stream function in custom.py. The example given is a sentence that is split, then streamed: response_chunks = "This is a response from a Flask server. Thank you for your message!".split(" ") response = Response(self.send_stream(response_chunks), mimetype="text/event-stream")

My LLM Chain is returned with a CallbackManager([StreamingStdOutCallbackHandler() stream from LLamaCpp.

The chunks from my LLM Chain processed via send_stream are of one character each, because they were not splitted first (as with the example), as they are provided in real time.

Thus my question is, how can I modify the send_stream and chat_stream functions to return my Llama_Cpp stream in real time?

Also a second subquestion: does DeepChat have a Discord group or such?

[OvidijusParsiunas]

Hi @calypset. The responses should be able to handle a stream of individual characters. E.g. if you change the split code into:

response_chunks = list("This is a response from a Flask server. Thank you for your message!")

It will work just fine.

Perhaps I may have misunderstood the problem, let me know, thanks!

In regards to your second question, currently there is no Discord server for Deep Chat. The reason for this is because I don't think the community for it is big enough just-yet and it is best to concentrate all of the trouble-shooting/suggestion related content in one place (GitHub) so all information is easy to find. I would personally love to create a server, but given my currently changing employment status - I don't want to commit to it in-case I may not be able to maintain it long-term. Hence when the dust will settle I will be able to see if I have the extra time for it. I am also currently working on another project - which is pretty much like Deep Chat but for all use-cases (not necessarily focused on AI), which should garner more attention to my chat components. Hence, once that is released I hope the interest/community will be much bigger to consider creating a Discord channel. In the meanwhile if you want to contact me directly, all of my main channels are on my profile, however I tend to check GitHub issues more often than these channels 😄

[OvidijusParsiunas]

Having looked at your statement further I think I understand what advice you are after. I am not an expert in Python, however I think what you can try to do is pass your stream (langChainStream) into send_stream like so:

response = Response(self.send_stream(langChainStream), mimetype="text/event-stream")

Then in send_stream, you can use the following code to observe your stream and respond with each individual character:

def send_stream(self, langChainStream):
  # write code to parse the responses from langChainStream and respond with
  yield f"data: {json.dumps({'text': f'{parsedCharacter} '})}\n\n"
  # end stream with
  # yield ""

You can end the stream with yield ""

Let me know if this works for you. If not, could you perhaps share more details on what code you are using. Thankyou!

[calypset]

Thank you I will try it! Thanks for the quick reply. List works for the time being. Let us know on your upcoming project(s).

[OvidijusParsiunas]

I'll close this issue since the problem has been given a solution, however feel free to comment if anything else comes up.

In regards to the other project, it is called active-chat and is in active development so it is unusable at the moment. It should hopefully get its first release sometime next month 🚀