boxcar for vector search

[francis]

Now that we have embedded vector search, add a boxcar that scans search results and gives a coherent summary.

Ideally, the class would be Boxcars::VectorSearch

@jaigouk - your new additions will enable this. I want to accept the default in-memory vector store that you provided or external (Posgresql, Redis, Pinecone, ...) stores.

[mculp]

Can take a cue from AutoGPT. Here's their setup: https://github.com/Significant-Gravitas/Auto-GPT/tree/master/autogpt/memory

[francis]

👍 Great idea @mculp - I will study this approach as it solves another problem I was considering too.

[jaigouk]

I will check this in this weekend @francis

observations

• some people want to use auto-gpt with more configurable options like langchain.
• it seems that langchain has more structure. I guess that is because of the nature of the library.
• probably i should rename the module name from embeddings to vectorstores

potential tasks

• [x] rename the module name from Embeddings to VectorStores
• [x] add default memory by referencing vectorstores/memory in langchain
• [x] refine hnswlib search to have the same logic as in_memory based search
• [x] add VectorSearch class and accept VectorStores when intializing or calling
• [x] example code

• [x] fix bugs

  update: I wanted to use tensorflow as another option like langchain does but the tensorflow gem is not yet ready for it.

gpt4all has a ticket for embeddings but it is still open.

[francis]

I like your thinking here @jaigouk

[jaigouk]

@francis https://github.com/BoxcarsAI/boxcars/pull/74 is using Boxcars::VectorStore::Document for building vectore stores and search results and Boxcars::VectorStore::Document has content, embedding and metadata. Can we close this ticket?

[jaigouk]

can we mark this issue as done? https://github.com/BoxcarsAI/boxcars/pull/92 is merged. @francis