search

BitFunnel / Workbench

Java and Lucene based tools for BitFunnel corpus preparation
http://bitfunnel.org
MIT License
19 stars 4 forks source link

Possibly spurious single letters #13

Closed danluu closed 7 years ago

danluu commented 7 years ago

A large fraction of documents have single letter terms. This looks pretty suspicious.

be6572d261265461,1,1,0.0380624,s
4b2cb04c8105642d,1,1,0.0299786,c
14b93e0f59ea8e65,1,1,0.0286755,b
MikeHopcroft commented 7 years ago

Individual letters are common in the references section. See, for example, https://en.wikipedia.org/?curid=39. Reference number 6 (for Tom Markvart; Luis CastaŁżer (2003). Practical Handbook of Photovoltaics: Fundamentals and Applications.) has a, b, c, d, and e.