Closed PxYu closed 2 years ago
The clueweb dataset directory, if it might be helpful.
ls ~/.ir_datasets/clueweb09/corpus/
checksums ClueWeb09_Chinese_3 ClueWeb09_English_2 ClueWeb09_English_6 ClueWeb09_French_1 ClueWeb09_Japanese_2 ClueWeb09_Spanish_2
ClueWeb09_Arabic_1 ClueWeb09_Chinese_4 ClueWeb09_English_3 ClueWeb09_English_7 ClueWeb09_German_1 ClueWeb09_Korean_1 record_counts
ClueWeb09_Chinese_1 ClueWeb09_English_1 ClueWeb09_English_4 ClueWeb09_English_8 ClueWeb09_Italian_1 ClueWeb09_Portuguese_1
ClueWeb09_Chinese_2 ClueWeb09_English_10 ClueWeb09_English_5 ClueWeb09_English_9 ClueWeb09_Japanese_1 ClueWeb09_Spanish_1
Folders other than checksums and record_counts are created using soft links.
Thanks for reporting. The above code works for me, but let's try to get to the bottom of what's going on here.
Can you run md5sum ~/.ir_datasets/clueweb09/corpus/ClueWeb09_English_1/en0000/00.warc.gz
(the file containing the above document). My version is 82cd52301030a50fdd6fa6e198bb6a07
, which matches what's in the checksums file (head ~/.ir_datasets/clueweb09/corpus/checksums//ClueWeb09_English_1_checksums.md5 -n1
).
My version is e200a39790702507e7f3b91eb5d93515
.
Gotcha, thanks. So this means that the zlib checkpoints will not work for performing fast lookups -- they are dependent on the files being compressed the same way.
This is a case we hadn't considered when designing this setup for working with ClueWeb. I think there are a couple of options:
What do you think? Is it possible that you have a version of ClueWeb that matches the official checksums somewhere?
cc @andrewyates
Thanks for the explanation!
I don't know why our files do not match the official checksums. The files have been on our server since 2009. I will contact our system administrator to see if I can get a more original copy.
In case that doesn't work, I might go with option 1. My use case is only document lookup (for around 60k documents). Is there something in the repo that could help this process?
There's nothing built into ir_datasets right now to do this. But here's a workaround for you: you can replace the checkpoint files with empty ones. Then it will always traverse the entire file (up to the requested documents) when performing lookups.
touch template
lz4c template
for f in ~/.ir_datasets/clueweb09/corpus.chk/*/*/*.chk.lz4 ; do cp template.lz4 $f ; done
Then you should be able to run the code from above no problem:
import ir_datasets
dataset = ir_datasets.load("clueweb09/en")
docstore = dataset.docs_store()
docid = "clueweb09-en0000-00-05965"
tmp = docstore.get(docid)
This seems to work fine for me-- let me know if it works for you!
Also- as a hint, use docstore.get_many
or docstore.get_many_iter
to avoid traversing the same file multiple times when a file contains multiple documents.
I'm still debating whether this is functionality we should build into ir_datasets or not.
error message:
FileNotFoundError: [Errno 2] No such file or directory: '/home/username/.ir_datasets/clueweb09/corpus.chk/ClueWeb09_English_10/en0131/68.warc.gz.chk.lz4'
Huh, is there anything in ~/.ir_datasets/clueweb09/corpus.chk/ClueWeb09_English_10/en0131/
? The for f in ~/.ir_datasets
code above should have taken care of replacing all the files it was expecting.
Weird that 68.warc.gz.chk.lz4
wasn't in ~/.ir_datasets/clueweb09/corpus.chk/ClueWeb09_English_10/en0131/
originally. I just copied the empty template there and this problem went away. But now it says
Traceback (most recent call last):
File "acquire_document_texts.py", line 28, in <module>
tmp = docstore.get(docid)
File "/home/username/miniconda3/lib/python3.8/site-packages/ir_datasets/indices/base.py", line 13, in get
raise KeyError(f'doc_id={doc_id} not found')
KeyError: 'doc_id=clueweb09-en0000-00-05965 not found'
Seems like it's not traversing the file when the checkpoint is empty.
Weird. It's hard for me to know if it's a problem with your version of the source files or the ir_datasets code somehow.
I wonder if the source file is substantially different in size (suggesting some documents are missing). Can you run:
ls -l ~/.ir_datasets/clueweb09/corpus/ClueWeb09_English_1/en0000/00.warc.gz
(mine is 168967171, ~162M)
Maybe also see if the document is in the source file directly like this?
zcat ~/.ir_datasets/clueweb09/corpus/ClueWeb09_English_1/en0000/00.warc.gz | grep -a clueweb09-en0000-00-05965
Finally, you could try the patch I just made to the branch https://github.com/allenai/ir_datasets/tree/issue-134, in case it's a problem with the checkpoint template file workaround. You'll need to set the environment variable IR_DATASETS_CW_SKIP_CHK=1
.
The patch works for me.
My file is slightly smaller than your version, but the documents were all there. Not sure why though...
Anyway, thank you for being very patient with me! I got no further questions here, so feel free to close this issue.
No problem-- I'm glad it's working. Sorry that the template workaround was more of a hassle than it was worth.
I'll close the issue, but can you keep me posted if you find anything about the different compression? It would be helpful to know if there are two "official" versions of the corpus going around, in which case we could provide built-in support for both versions.
Describe the bug Hi. I have trouble using the docstore.get() function. About half the documents that I am interested in return the following message. Some document ids work just fine, which makes this very confusing...
This is how I initiate the docstore:
Affected dataset(s)
Clueweb09.
To Reproduce
This snippet would result in the aforementioned message. I do not know what is the issue. Is it that my clueweb files are corrupted? I tried to clear cache for docstore, didn't help.
Any hint or help would be appreciated. Thanks!