Support for the sparse embeddings - Githubissues

langchain-ai / langchain-postgres

LangChain abstractions backed by Postgres Backend

MIT License

66 stars 22 forks source link

Support for the sparse embeddings #71

Open magaton opened 3 weeks ago

magaton commented 3 weeks ago

The latest pgvector version supports sparsevec. However, langchain's PGVector supports only one embeddings column in langchain_pg_embedding table. It would be great to have a sparse_embedding column and sparse_embedding field in PGVector.

I have considered the alternative and that is to have 2 PGVector stores, 1 for dense and 1 for sparse vectors. However there are 2 problems with that:

PGVector has hardcoded table names for collection and embeddings
I would like to leverage excellent langchain indexer with SQL manager.

gecBurton commented 1 week ago

hi @magaton I would be interested in collaborating on this, I would also like some kind of full-text/dense feature https://github.com/langchain-ai/langchain-postgres/issues/61