search

hegelai / prompttools

Open-source tools for prompt testing and experimentation, with support for both LLMs (e.g. OpenAI, LLaMA) and vector databases (e.g. Chroma, Weaviate, LanceDB).
http://prompttools.readthedocs.io
Apache License 2.0
2.65k stars 230 forks source link

Add support for LanceDB #48

Closed NivekT closed 1 year ago

NivekT commented 1 year ago

🚀 The feature

LanceDB is a developer-friendly, serverless vector database.

Motivation, pitch

We can integrate with its APIs to allow experimentation on various configurations and look at its performance.

Alternatives

No response

Additional context

You can have a look at the Chroma or Weaviate for inspiration. If you would like to work on this, comment and let us know! We will be more than happy to support you.

HashemAlsaket commented 1 year ago

Heads up- need to have Rust installed. It's a dependency for tantivy-py which is a dependency for LanceDB.

NivekT commented 1 year ago

This is being worked on by myself and @AyushExel from LanceDB

AyushExel commented 1 year ago

Thanks for the PR @HashemAlsaket @NivekT tantivy is not a compulsory requirement. Lancedb doesn't require any extra setup or auth other than pip install lancedb

HashemAlsaket commented 1 year ago

I was unable to search a table without tantivy: image

AyushExel commented 1 year ago

You're using FTS. FTS requires tantivy, vector search doesn't. FTS is an add-on feature for now

HashemAlsaket commented 1 year ago

Ah ok, thanks for the clarification!

NivekT commented 1 year ago

Added in #71