tholor
commented
3 years ago Hey @vinchg ,
Thanks for raising this issue. It's highly relevant.
So in general: for larger datasets like this, we would recommend not using SQLite but rather Postgres. You can easily spin one up via a Docker:
docker run --name haystack-postgres -p 5432:5432 -e POSTGRES_PASSWORD=password -d postgres
docker exec -it haystack-postgres psql -U postgres -c "CREATE DATABASE haystack;"and then connect in Python via:
document_store = FAISSDocumentStore(sql_url="postgresql://postgres:password@localhost:5432/haystack",
faiss_index_factory_str=index_type)However, this will probably only increase memory efficiency but not resolve the underlying problem.
I totally agree that we should add some batch functionality to update_embeddings and a tqdm progress bar.
The call to query.all() is what's causing the issue.
Do you have already any embeddings in the document store or is that a fresh one?
Is there an alternative way with dealing with large datasets like this?
As a temporary workaround, you could generate the embeddings in batches yourself before calling write_documents() and attach them to your documents before writing to the DocumentStore. Rough sketch:
from haystack import Document
dicts = [{"text": "some text"}, ...]
docs = [Document.from_dict(d) for d in dicts]
# get the embedding for a batch of docs
batch_docs = docs[:32]
batch_emb = retriever.embed_passages(batch_docs)
# attach embeddings to the docs
for emb, doc in zip(batch_emb, batch_docs):
doc.embedding = emb
...
# later: write everything to the doc store
doc_store.write_documents(all_docs)Again, we should in any case add the batch functionality soon in update_embeddings() to simplify usage here ...
@lalitpagaria Would this maybe be something that is of interest to you and you would like to work on? If not, @tanaysoni can take care of it.
lalitpagaria
vinchg
brandenchan
Question As mentioned in the title, I'm trying to update the embeddings for Faiss (HNSW) with a DPR retriever on a large dataset (~15 million documents). Following the tutorial steps, I'm writing the documents to a local sqlite db (~15 gb .db). However, calling update_embeddings uses all my RAM (64 GB) and all my swap (64 GB) and proceeds to run out of memory after running for hours. I'm fairly certain the line that's consuming so much memory is: https://github.com/deepset-ai/haystack/blob/3f81c93f36519ab78213f145c699ce7df2c4ddf8/haystack/document_store/faiss.py#L158
More specifically: https://github.com/deepset-ai/haystack/blob/3f81c93f36519ab78213f145c699ce7df2c4ddf8/haystack/document_store/sql.py#L124 The call to query.all() is what's causing the issue.
I'm not very familiar with SQLAlchemy and SQLite in general but it seems that there is some inefficient usage of space when querying the DB. Is there an alternative way with dealing with large datasets like this? It would be preferable if there was a batch option to update_embeddings so that a group of embeddings would be flushed to disk before proceeding.
On a side note, some type of progress indicator or option to enable one when calling write_documents or update_embeddings would be useful (considering these operations can take an hour or more).