Open AbeHandler opened 8 years ago
the LDC NYT definitely has dupes. here's my shingling code i've used before on the union of gigaword and NYT which seemed ok on spot checks:
https://github.com/brendano/OConnor_IREvents_ACL2013/blob/master/code/preproc/docdedup.py
We use a simple form of shingling (ch. 3, Rajaraman and Ullman, 2011): represent a document signature as its J = 5 lowercased bigrams with the lowest hash values, and reject a document with a signature that has been seen before within the same month. J was manually tuned, as it affects the precision/recall tradeoff.
see aug 20 1994, "Refugee Weapon Has Been Wielded Before" in cuba corpus
Not sure where this is coming from (penn LDC?). The dupes are as high up as /corpora/country/processed/all.anno_plus. For now now, just doing $sort all.anno_plus | uniq to fix