search

spacemanidol / MSMARCO

Utilities, Baselines, Statistics and Descriptions Related to the MSMARCO DATASET
MIT License
189 stars 41 forks source link

[encoding,Â] top1000.dev.tsv #26

Closed Albert-Ma closed 5 years ago

Albert-Ma commented 5 years ago

I read the content from top1000.dev.tsv, but met many encoding problems.

For example:

Right text in queries.tsv:what complication is a potential danger associated with continuous iv infusions?

Problem text in top1000.dev.tsv: what complication is a potential danger associated with continuous iv infusions?

There are many places appear this problem which text contain  in top1000.dev.tsv.

Albert-Ma commented 5 years ago

@dfcf93

spacemanidol commented 5 years ago

Hey,

You are correct. We seem to have a bug in the topfiles. I will go ahead and update the files at some point but in the meantime each line also includes the QID and PID(Question ID) and (Passage ID) which you can join with the collection.tsv and the query files. Sorry for the delay.

Albert-Ma commented 5 years ago

Hi, And also found this problem in train.triples file, but not in collection. I think it's a encoding problem while parsing web document. Thanks for the reply.

spacemanidol commented 5 years ago

Just not to leave you in the dark I am going to update all the documents to fix this issue I am just currently away on paternity leave. I should be able to get to it mid may

Albert-Ma commented 5 years ago

Just not to leave you in the dark I am going to update all the documents to fix this issue I am just currently away on paternity leave. I should be able to get to it mid may

That's very kind of you.

spacemanidol commented 5 years ago

This has been fixed and uploaded for the top1000 dev and eval. The top1000 train and the train triples will follow shortly.

for context I used https://ftfy.readthedocs.io/en/latest/

import ftfy import sys

filename = sys.argv[1] with open(filename, 'r') as f: with open('cleaned' + filename, 'w') as w: for l in f: w.write(ftfy.fix_text(l))