error occurs at update_embeddings

3CE8D2BAC65BDD6AA9 commented 1 year ago

I have been trying https://github.com/weaviate/weaviate-examples/blob/main/harrypotter-qa-haystack-weaviate/JUPYTER-HarryPotter-QA-Haystack-Weaviate.ipynb a few times at my local machine.

My code is:

from haystack.utils

import clean_wiki_text import pandas as pd

harry = pd.read_csv("https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/harry_potter_wiki.csv")

dicts = []

for ix, row in harry.iterrows(): dic = {
    'content': clean_wiki_text(row.text),
    'meta': {
        'name': row['name'],
        'url': row.url
    }
}
dicts.append(dic)
from haystack.utils import launch_weaviate launch_weaviate() from haystack.document_stores import WeaviateDocumentStore document_store = WeaviateDocumentStore() # assumes Weaviate is running on http://localhost:8080 document_store.write_documents(documents=dicts, batch_size=100) from haystack.nodes import EmbeddingRetriever

retriever = EmbeddingRetriever(document_store=document_store, model_format="sentence_transformers", embedding_model="sentence-transformers/multi-qa-mpnet-base-dot-v1",) document_store.update_embeddings(retriever) from haystack.nodes import FARMReader reader = FARMReader(model_name_or_path="deepset/tinyroberta-squad2", use_gpu=True)

everything went smooth until this line:

document_store.update_embeddings(retriever)

The error message is:

Exception has occurred: WeaviateDocumentStoreError Query results contain errors: [{'locations': [{'column': 6, 'line': 1}], 'message': 'explorer: list class: search: invalid pagination params: query maximum results exceeded', 'path': ['Get', 'Document']}] File "/home/ky/weaviate/harrypotter.py", line 26, in document_store.update_embeddings(retriever) haystack.document_stores.weaviate.WeaviateDocumentStoreError: Query results contain errors: [{'locations': [{'column': 6, 'line': 1}], 'message': 'explorer: list class: search: invalid pagination params: query maximum results exceeded', 'path': ['Get', 'Document']}]

Any idea how to resolve this?

Heucles commented 1 year ago

I am also facing the same issue

Heucles commented 1 year ago

Hey @3CE8D2BAC65BDD6AA9 I was able to get it to work increasing the value of QUERY_MAXIMUM_RESULTS env, the default is 10K, for me doubling it made it work.

I've also was able to find a couple of threads discussing the issue and I'm currently looking for a better solution, here are the links bellow: https://github.com/weaviate/weaviate/issues/1947 https://github.com/deepset-ai/haystack/issues/2517 https://github.com/deepset-ai/haystack/issues/3390

Hope those will help you.

Heucles

weaviate / weaviate-examples

error occurs at update_embeddings #56