search

microsoft / kernel-memory

RAG architecture: index and query any data using LLM and natural language, track sources, show citations, asynchronous memory patterns.
https://microsoft.github.io/kernel-memory
MIT License
1.6k stars 307 forks source link

[QUESTION] Comparison of memory sources #149

Closed DrEight closed 12 months ago

DrEight commented 1 year ago

I don't understand the difference of using Azure Blob Storage vs Cognitive Search from the KM point of view. In my tests I enabled both by mistakes and KM was working, but it was not clear to me which memory source was using. Can you please help me to understand the benefit of using one of more memory source and the use case scenarios ?

dlucr commented 1 year ago

Azure Blob Storage is used to keep a copy of files, e.g. the file you upload, and intermediary files like the text extracted, and some JSON files. Data here is also useful to generate "citations", when answering a question, pointing the user to the document files used to answer.

Azure Cognitive Search is used to store "search indexes" about the data uploaded. We currently use vector indexes, and eventually also semantic search. This service is where embeddings are stored, and how relevant information is found in order to answer a question.

DrEight commented 1 year ago

Thank for the clarification. If I understand correctly, to give the best service I have to use both. Correct ?

dluc commented 1 year ago

Yes using both a file storage and a vector storage will give you the best experience, bear in mind costs though and considering how big your solution will grow over time.

dluc commented 12 months ago

It looks like the question has been addressed. If you come across any more issues or have additional questions, feel free to reopen this issue.