search

facebookresearch / DPR

Dense Passage Retriever - is a set of tools and models for open domain Q&A task.
Other
1.73k stars 304 forks source link

NotImplementedError when using dense_retriever.py with HNSW index #152

Closed ikuyamada closed 3 years ago

ikuyamada commented 3 years ago

Hi,

When running dense_retriever.py with indexer=hnsw, I got NotImplementedError with the following stacktrace:

Error executing job with overrides: ['model_file=/home/ikuya/projects/DPR/downloads/checkpoint/retriever/single-adv-hn/nq/bert-base-encoder.cp', 'qa_dataset=nq_test', 'ctx_datatsets=[dpr_wiki]', 'encoded_ctx_files=["/home/ikuya/projects/DPR/downloads/data/retriever_results/nq/single-adv-hn/wikipedia_passages_*"]', 'out_file=out_hnsw.json', 'indexer=hnsw']
Traceback (most recent call last):
  File "dense_retriever.py", line 402, in main
    input_paths, index_buffer_sz, path_id_prefixes=path_id_prefixes
  File "dense_retriever.py", line 166, in index_encoded_data
    self.index.index_data(buffer)
  File "/home/ikuya/projects/DPR/dpr/indexer/faiss_indexers.py", line 33, in index_data
    raise NotImplementedError
NotImplementedError

I cloned the master branch and executed the following command:

python dense_retriever.py model_file=/home/ikuya/projects/DPR/downloads/checkpoint/retriever/single-adv-hn/nq/bert-base-encoder.cp qa_dataset=nq_test ctx_datatsets=[dpr_wiki] encoded_ctx_files=[\"/home/ikuya/projects/DPR/downloads/data/retriever_results/nq/single-adv-hn/wikipedia_passages_*\"] out_file=out_hnsw.json indexer=hnsw
vlad-karpukhin commented 3 years ago

Just fixed https://github.com/facebookresearch/DPR/commit/8bacf08fa9be0677f5b9ef973b1cc181cc66a2af Please update and try again.

ikuyamada commented 3 years ago

Thank you for your prompt reply! The error has been fixed now.