search

deepset-ai / haystack-integrations

🚀 A list of Haystack Integrations, maintained by the community or deepset.
42 stars 52 forks source link

Add lancedb-haystack to integrations. #238

Closed alanmeeson closed 1 month ago

alanmeeson commented 1 month ago

This PR adds the LanceDB Haystack Document Store from https://github.com/alanmeeson/lancedb-haystack.

Notes from Discord discussion: There are some interesting quirks due to the nature of LanceDB; you need to specify a schema for your metadata so that it can be turned into a pyarrow struct (and thus be searchable). Also the precision of the embedding vectors seems to be limited to an error of around 10^-7 compared to it's python form - feels like some sort of difference between the float representations.

Please take a look when you have time, and let me know if I should change anything.

Thanks!

alanmeeson commented 1 month ago

I've updated the pip install block to be tagged as shell, got in touch with the LanceDB team and sourced the official 'light on dark' logo, and added a link to the new API docs.