qdrant / qdrant-haystack

An integration of Qdrant ANN vector database backend with Haystack
Apache License 2.0
43 stars 12 forks source link

Loading multiple pipelines from yaml creates different instances of QdranDocumentStore for each pipeline #17

Closed RajanMehta closed 1 year ago

RajanMehta commented 1 year ago

I have two pipelines defined in YAML: 1. query_pipeline 2. indexing_pipeline.

indexing_pipeline saves data to the document store. Now, when I run the query_pipeline, it doesn't have access to the indexed data.

Seems like "Store" component of query_pipeline (retriever) and "Store" component of indexing_pipeline are different instances.

I was able to fix the issue with this hack:

document_store = pipelines["indexing_pipeline"].get_document_store()
retriever = pipelines["query_pipeline"].get_node("Retriever")
retriever.document_store = document_store

Notebook to reproduce the issue: https://colab.research.google.com/drive/1MTk2VEmi3LjkavpKEhG9IHIAUeNFGOnJ?usp=sharing

RajanMehta commented 1 year ago

I guess this is only happening with ":memory:" location. The pipelines are working fine with "localhost"