search

Azure-Samples / rag-postgres-openai-python

A RAG app to ask questions about rows in a database table. Deployable on Azure Container Apps with PostgreSQL Flexible Server.
MIT License
106 stars 17 forks source link

Add streaming #27

Open pamelafox opened 1 month ago

pamelafox commented 1 month ago

Related to #26, customers would like streaming. Can use Chat Protocol SDK

john0isaac commented 2 weeks ago

I would like to work on that!

pamelafox commented 1 week ago

Awesome, I will assign it to you!

Since we're using the Chat Protocol SDK, you'll want to adhere the backend to that and then use the streaming method on the frontend. The azure-search-openai-demo already conforms to the protocol backend, so you can take a similar approach there, adding a /stream endpoint that returns json-lines.

The azure-search-openai-demo frontend doesn't yet use the Chat Protocol SDK in the main branch, but I did bring it in in a branch here: https://github.com/Azure-Samples/azure-search-openai-demo/compare/main...pamelafox:azure-search-openai-demo:chatprotocolsdk#diff-506debba46b93087dc46a916384e56392808bcc02a99d9291557f3e674d4ad6cR177

You can also compare that with the protocol SDK react sample here: https://github.com/microsoft/ai-chat-protocol/tree/main/samples/frontend/js/react

However, azure-search-openai-demo implements a setTimeout that works better for token-by-token streaming than the sample in that repo.

Let me know if any questions along the way.

pamelafox commented 1 week ago

Oh, and confession: there are no true tests for this repo yet. A bit embarassed to admit that since you saw my workshop yesterday on testing! So you will need to manual testing, or be the first to add tests.

john0isaac commented 1 week ago

Thanks for the explanation! ❤️ I was following you on GitHub when you ported to chat protocol. So don't worry about that.

I finally managed to add streaming to my sample yesterday that's why I wanted to add it here too. https://github.com/Azure-Samples/Cosmic-Food-RAG-app/pull/39

I didn't show up here earlier as I just figured this out yesterday.

I will work on some tests too.