Chore - refine VectorSearch

BoxcarsAI / boxcars

Building applications with composability using Boxcars with LLM's. Inspired by LangChain.

MIT License

430 stars 39 forks source link

Chore - refine VectorSearch #74

Closed jaigouk closed 1 year ago

jaigouk commented 1 year ago

changes

moved common methods into lib/boxcars/vector_store.rb
Boxcars::VectorStore::Document has content, embedding and metadata
use Boxcars::VectorStore::Document for building vectore stores and search results
added Boxcars::VectorStore::InMemory::BuildFromFiles
removed lib/boxcars/vector_store/similarity_search.rb use VectorSearch

why

need to hold documents with new method and call for search multiple times with different query
consistent structure for search classes
~~create query vector within SimilaritySearch~~
revmoed SimilaritySearch and updated VectorSearch
resolves part of https://github.com/BoxcarsAI/boxcars/issues/60

jaigouk commented 1 year ago

we have VectorSearch class. do we need to keep Boxcars::VectorStore::SimilaritySearch? @francis

jaigouk commented 1 year ago

extra clean up would be adding the metadata for hnswlib search. or use the same structure as in_memory search. I am not sure which one is better.

francis commented 1 year ago

@jaigouk - we probably don't need Boxcars::VectorStore::SimilaritySearch now

francis commented 1 year ago

I'm not going to merge until you give the go ahead @jaigouk

jaigouk commented 1 year ago

@francis I think it is ready to be reviewed. Sorry about too many files. I wanted to be consistent about using the Document class for having metadata, etc.

I updated gem version number to 0.2.12

For notebooks/embeddings/embeddings_example.ipynb, I installed https://github.com/SciRuby/iruby and second section for InMemory vectore store codes were not able to load newly added file. I was not able to figure out the reason.

francis commented 1 year ago

I’ll take a look.