Open talk2much opened 4 months ago
I have all the files I need for the index except ivfpq.100.faiss
Hi, It seems the generated files also don't have the ivfpq.faiss file? so, this means the indexing process didn't succeed. Normally, for creating the Robust04 ColBERT& ColBERT-PRF indices, we consider splitting the long documents of Robust04 into smaller passages. Maybe you can try the following code:
from pyterrier_colbert.indexing import ColBERTIndexer
index_root="/some/path"
index_name="index_name"
# default 150, stride 75
indexer = pt.text.sliding(text_attr="text", prepend_title=False)>> ColBERTIndexer("/path/to/colbert.dnn",index_root, index_name, chunksize=20)
indexer.index(pt.get_dataset("irds:trec-robust04").get_corpus_iter())
Hi, It seems the generated files also don't have the ivfpq.faiss file? so, this means the indexing process didn't succeed. Normally, for creating the Robust04 ColBERT& ColBERT-PRF indices, we consider splitting the long documents of Robust04 into smaller passages. Maybe you can try the following code:
from pyterrier_colbert.indexing import ColBERTIndexer index_root="/some/path" index_name="index_name" # default 150, stride 75 indexer = pt.text.sliding(text_attr="text", prepend_title=False)>> ColBERTIndexer("/path/to/colbert.dnn",index_root, index_name, chunksize=20) indexer.index(pt.get_dataset("irds:trec-robust04").get_corpus_iter())
I get an error when I don't change the code
Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/site-packages/colbert/indexing/faiss.py", line 92, in _loader_thread
sub_collection = [load_index_part(filename) for filename in filenames if filename is not None]
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/site-packages/colbert/indexing/faiss.py", line 92, in <listcomp>
sub_collection = [load_index_part(filename) for filename in filenames if filename is not None]
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/site-packages/colbert/indexing/index_manager.py", line 17, in load_index_part
part = torch.load(filename)
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/site-packages/torch/serialization.py", line 1005, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/home/yujy/anaconda3/envs/colbert/lib/python3.8/site-packages/torch/serialization.py", line 457, in __init__
super().__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: PytorchStreamReader failed reading zip archive: not a ZIP archive
It looks like the thread won't start, right
The location of this error is
[4月 25, 10:52:31] #> Indexing the vectors...
[4月 25, 10:52:31] #> Loading ('/home/yujy/code/Colbert_PRF/index/robust04_index/0.pt', '/home/yujy/code/Colbert_PRF/index/robust04_index/1.pt', '/home/yujy/code/Colbert_PRF/index/robust04_index/2.pt') (from queue)...
Exception in thread Thread-2:
Traceback (most recent call last):
Hi, It seems the generated files also don't have the ivfpq.faiss file? so, this means the indexing process didn't succeed. Normally, for creating the Robust04 ColBERT& ColBERT-PRF indices, we consider splitting the long documents of Robust04 into smaller passages. Maybe you can try the following code:
from pyterrier_colbert.indexing import ColBERTIndexer index_root="/some/path" index_name="index_name" # default 150, stride 75 indexer = pt.text.sliding(text_attr="text", prepend_title=False)>> ColBERTIndexer("/path/to/colbert.dnn",index_root, index_name, chunksize=20) indexer.index(pt.get_dataset("irds:trec-robust04").get_corpus_iter())
After debugging I found that the reason for the error is that all my pt files torch are not loading, so it will report an error:
RuntimeError: PytorchStreamReader failed reading zip archive: not a ZIP archive
And what does Google say when you search for this error message?
Hello. Now I get the most of the need when the index files such as doclens.10.json,docnos.pkl.gz files But in the last step write in ivfpq.100.faiss file failed So I want to use the obtained file to write in the ivfpq.100.faiss file. My code is as follows:
But it didn't work out and got stuck: