search

OvidijusParsiunas / deep-chat

Fully customizable AI chatbot component for your website
https://deepchat.dev
MIT License
1.27k stars 176 forks source link

Media not displaying when simulating a stream #45

Closed succinct0274 closed 7 months ago

succinct0274 commented 7 months ago

Hi Ovidijus, first of all, thank you for the great work you have made. It is a really nice project. I recently took this to make a chatbot with Langchain. I noticed that images will not be shown if simulated stream is enabled. I have spent some time looking at the source code. It seems that the static method Stream.simulate will not consider any images in the response. Will there be any future enhancement on this?

Anyway, thanks in advance.

OvidijusParsiunas commented 7 months ago

Hi @succinct0274. Currently streaming supports text only. I have been thinking about improving this API to allow other data types. Could you tell me more about your use-case and would the stream with an image look in the chat:

Would you want an image to appear alongside text in the same bubble? So there would be streamed text, then an image would appear, then potentially more streamed text after it? Or perhaps would you want the image to be in a different bubble? Would there be multiple images?

Thanks!

succinct0274 commented 7 months ago

Hi @OvidijusParsiunas, my backend server will plot the charts and send back as the response along with the text, the response format would be like

{
  "text": <text response from the open ai>,
  "images": [ { "content": <binary image encoded into base64 string>, "content_type": <mime_type string>},  ...]
}

the handler at the frontend side will receive the response and further convert into the type MessageContent, which is specified in the documentation.

Mya thought would be showing images after text has been streamed. I think showing in the same bubble is fine.

Thanks in advance.

succinct0274 commented 7 months ago

yes, I am mentioning the simulated stream

OvidijusParsiunas commented 7 months ago

Sorry I misread the issue. I should be able to fix this. In terms of user experiences I think the media/images should appear first and the text second so as-to not constantly move the image down as the text is populated. Will this work for you?

succinct0274 commented 7 months ago

Sure, thanks for your help

OvidijusParsiunas commented 7 months ago

I have written the initial code for this and will be able to bundle it with the next release that is going to happen in the next couple of days. I will update this thread on any news.

OvidijusParsiunas commented 7 months ago

Hey, sorry for the hold up as there have been a lot of changes happening in the component code. This feature will be released in the main package later this week, but you can use it early in our deep-chat-dev and deep-chat-react-dev packages version 9.0.66. These packages work the same way it's just their names are different. Let me know if you have any issues. I will update this thread when the main package is released.

OvidijusParsiunas commented 7 months ago

Hi. The functionality discussed above has now been released in Deep Chat version 1.4.4. Please read the release notes for more information.

To note, simulation functionality has been changed to populate by character rather than word. So the recommended default number value is 6, but you can ofcourse change it to your preference.

I will be closing this issue, but if you experience problems relating to streaming feel free to comment below or raise a new issue for anything else. Thanks!