[FEATURE] Querying live data without vector databases or building an LLM app without vector databases

[Boburmirzo]

Is your feature request related to a problem? Please describe.

I see there are many use cases of vector databases provided on OpenAI Cookbook examples such as for powering semantic search, question answering, and recommendations. But I don’t see any examples with other storage other than vector databases for these LLM applications.

Vector databases come with costs like increased prep work, infrastructure, and complexity. Keeping source and vectors in sync is painful. Also, it is even harder if the underlined input data is changing over time and requires re-indexing.

Describe the solution you'd like

I found that there are already some solutions on GitHub that might address the underlying problem - a solution without a vector database. I tried an open-source LLM App (the link below to the repo) which has taken care of both the document pipeline and a vector index and keeping things in sync.

Would you be interested in providing a showcase and guide on how to use it? We can help with publishing a new Notebook under the repo.

Additional context

Visit the project repo: https://github.com/pathwaycom/llm-app See how it works in this video: https://www.youtube.com/watch?v=kcrJSk00duw or Read the article. https://pathway.com/developers/showcases/llm-app-pathway/

Relevant discussion on Linkedin and Twitter: https://twitter.com/mistercrunch/status/1681028417716903941?s=20 https://www.linkedin.com/posts/anupsurendran_llm-vectorsearch-vectordatabase-activity-7090376720104534016-STEu?utm_source=share&utm_medium=member_desktop

### Tasks
- [ ] Add a notebooke guide for LLM App usage

[Boburmirzo]

@ted-at-openai Hi Ted,

Could you provide any feedback on this, please? If you think this can be beneficial for the OpenAI community, I can help to contribute to a new guide for the repo.

[Yosoyun]

I loved it .

On Wed, 2 Aug 2023 at 3:58 PM Bobur Umurzokov @.***> wrote:

@ted-at-openai https://github.com/ted-at-openai Hi Ted,

Could you provide any feedback on this, please? If you think this can be beneficial for the OpenAI community, I can help to contribute to a new guide for the repo.

— Reply to this email directly, view it on GitHub https://github.com/openai/openai-cookbook/issues/618#issuecomment-1661957726, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZRAIWK4TE6JE67UD3H6KNLXTITWPANCNFSM6AAAAAA272LCCQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- 9010593514

[anupsurendran]

@Boburmirzo , what were you planning to do as a showcase? What would be simpler to create with a vector index-powered LLM App than a vector database-powered LLM App?

[Boburmirzo]

@isafulf @logankilpatrick Any input would you like to provide?

I am happy to contribute to the guide if you make confirmation:)

[Boburmirzo]

@Boburmirzo , what were you planning to do as a showcase? What would be simpler to create with a vector index-powered LLM App than a vector database-powered LLM App?

Thank you for your questions! In the showcase, I would like to demonstrate how it is easy to build an LLM App in 30 lines of code. Like a chatbot answering questions about a certain product's documentation.

• Implement a simple API that responds in real-time to HTTP REST queries.
• Implement a real-time document indexing pipeline directly reading data from S3-compatible storage or local storage, without having to query a vector document database which is faster, less expensive, and less overhead on building/maintaining additional databases in your infrastructure. Pathway will do indexing on the generated embeddings automatically.

[zakharsmirnoff]

@Boburmirzo vectordb is not necessarily an overhead. For example, with chromadb you can achieve the same functionality as you requested in just 4 lines of code. Chroma is free, open-source and you have an option not to deploy it anywhere and store locally. Also, Langchain provides VectorIndex as far as I know.

Real-time document indexing pipeline from S3 storage is not always faster. If you have millions or billions of documents, that would be nearly unusable. The same is true if your user base is growing. Without a real database server, it would be very difficult to handle all the requests with minimal latency.

[Boburmirzo]

@zakharsmirnoff Thank you for commenting on this. I agree partially with your above arguments and the main intention of raising this issue was not to fully ignore vector databases but to give an option for developers to use vector indexes based on their use cases.

Both have pros and cons depending on where it is applied, In this article, the author explains well when to use Vector Databases vs Vector Indexes?

Vector Databases are useful when one or more of the following is true

You have a specialized need for working with vector data at scale You are creating a standalone purpose-built application for vectors You do not expect other types of use for your stored data in other types of applications.

Vector Indexes are useful when one or more of the following is true

You do not want to trust new technology for your data storage Your existing storage is easy to access from Python. Your similarity search is just one capability among other larger enterprise BI and database needs You need the ability to attach vectors to existing scalar records You need one unified way of dealing with pipelines for your data engineering team You need index and graph structures on the data to help with your LLM apps or tasks You need augmented output or augmented context coming from other sources You want to create rules from your corpus which can apply to your transactional data

[zakharsmirnoff]

@Boburmirzo thank you for this, I appreciate the detailed response. Almost everything makes sense, except for the scaling. I do believe that for small-to-medium projects it's actually doesn't really matter what technology you use: you can write everything Turbo Pascal and store your data in plain txt files.

The things are getting interesting when it comes to scaling. Basically, using vector indices is the same as using plain files (or in-memory or SQLite for your general-purpose db.

Though, to be honest, I'm not quite familiar with the proposed setup, probably it would work even for large apps, though I prefer to stick with the common db approach, as in many cases it doesn't bring your new overhead. There are less capable and more capable databases, so you are not really limited in your choice

[Boburmirzo]

@Boburmirzo thank you for this, I appreciate the detailed response. Almost everything makes sense, except for the scaling. I do believe that for small-to-medium projects it's actually doesn't really matter what technology you use: you can write everything Turbo Pascal and store your data in plain txt files.

The things are getting interesting when it comes to scaling. Basically, using vector indices is the same as using plain files (or in-memory or SQLite for your general-purpose db.

Though, to be honest, I'm not quite familiar with the proposed setup, probably it would work even for large apps, though I prefer to stick with the common db approach, as in many cases it doesn't bring your new overhead. There are less capable and more capable databases, so you are not really limited in your choice

Thank you for keeping the discussion engaging:) That's why I suggested creating a guide here to show examples of how to make real-time indexing with easy scaling capabilities. You can already find lots of articles saying that you do not need a dedicated vector database for doing vector search operations. As you said, you can still use your own data storage like an OLTP database without a special need for a vector database. The solution we are trying to propose is in the application layer not in the infrastructure level.

[Boburmirzo]

@Boburmirzo thank you for this, I appreciate the detailed response. Almost everything makes sense, except for the scaling. I do believe that for small-to-medium projects it's actually doesn't really matter what technology you use: you can write everything Turbo Pascal and store your data in plain txt files. The things are getting interesting when it comes to scaling. Basically, using vector indices is the same as using plain files (or in-memory or SQLite for your general-purpose db. Though, to be honest, I'm not quite familiar with the proposed setup, probably it would work even for large apps, though I prefer to stick with the common db approach, as in many cases it doesn't bring your new overhead. There are less capable and more capable databases, so you are not really limited in your choice

Thank you for keeping the discussion engaging:) That's why I suggested creating a guide here to show examples of how to make real-time indexing with easy scaling capabilities. You can already find lots of articles saying that you do not need a dedicated vector database for doing vector search operations. As you said, you can still use your own data storage like an OLTP database without a special need for a vector database. The solution we are trying to propose is in the application layer not in the infrastructure level.

If you want to have a discussion in the specific scaling scenario, let me know we can jump on a call and I can explain more about what our setup looks like.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 10 days with no activity.