Feature: Adding context with Embeddings and Pinecone

[ronakv]

Not sure if this is a feature request, but it might be nice to interface with a vector database (ex. Pinecone) with OpenAI embeddings and add some text to add context to some messages.

[enricoros]

This is super interesting. Can be used to relate/group conversations, or search messages as well. Would need to get some sort of API key from the user, so that storage can be private.

How would this feature look like? (UI/UX)

[z-btc]

Embeddings, as represented by vectors of floating-point numbers, measure the "relatedness" of text strings. These are super useful for ranking search results, clustering, classification, etc. Relatedness is measured by cosine similarity. If the cosine similarity between two vectors is close to 1, the vectors are highly similar and point in the same direction. In the case of text embeddings, a high cosine similarity between two embedding vectors indicates that the corresponding text strings are highly related.

This starter app uses embeddings to generate a vector representation of a document, and then uses vector search to find the most similar documents to the query. The results of the vector search are then used to construct a prompt for GPT-3, which is then used to generate a response. The response is then streamed to the user. Check out the Supabase blog posts on pgvector and OpenAI embeddings for more background.

taken this from https://github.com/gannonh/gpt3.5-turbo-pgvector

maybe a way to integrate with each other

[z-btc]

in case of UI/UX, I have tested the above and they take a list of URL's to take embeddings from and stores on Supabase and calls Supabase API

[sungkim11]

Can we have the option to click on a response that opens up embedding results, which is pandas data frame for example? There can be multiple results so we will need paginations and support for multiple columns in the data frame.

[enricoros]

@sungkim11 can you describe a bit more. Pandas is not something we have on a Typescript website. Happy to add, but would need more details on what's the ask. For instance:

• what's the embedding generator, what's the chunking function for the text, when do you compute/recompute them, etc.

[sungkim11]

This assumes I am using OpenAI API where python is used to control the prompts and typescript is used to display these prompts.

User enters a prompt. Python script searches embeddings using prompt. Python script returns both response and embedding search results as pandas dataframe fir example. Typescript displays a response. When a user clicks on a response and triggers more information on a response, right nav opens up providing information on response, displaying pandas dataframe in tabular format within a limited space with a pagination controls. It could take up half the windows space.

I guess we can do a wholething in typescript too.

Thanks!

[sungkim11]

I guess I did not answer your question.

• This assumes embedding database exists - e.g., pinecone
• A query will be made to the embedding database, this entails the prompt will be encoded into a vector using openai embedding service (text-embedding-ada-002).
• Query results will be returned from the vector database
• Query results will need to be decoded using openai embedding service (text-embedding-ada-002).
• Query results will be displayed as an option to a response prompt.

[enricoros]

Thinking of closing this issue. Looking at this from now: s the purpose to have a RAG system in big-AGI?

If so, I can rename this issue and track it as "RAG", so a user could opt-in on a per-chat (or per-persona?) basis.

As far as LangChain and Pinecone - embeddings can be stored in any manner, and Langchain is library that's very large for the benefit it would provide.

[enricoros]

Closed for deprecation - superseded by RAG mechanisms.