Strange TF-IDF description

JetBrains-Research / pubtrends

Scientific literature explorer. Runs a Pubmed or Semantic Scholar search and allows user to explore high-level structure of result papers

Apache License 2.0

36 stars 2 forks source link

Strange TF-IDF description #209

Closed olegs closed 4 years ago

olegs commented 4 years ago

Search query human aging at Pubmed. Screenshot 2020-01-22 at 22 12 38 Click on word Alzheimers, you'll get only a single paper.

olegs commented 4 years ago

Alzheimer's became ['alzheimer'] during tokenisation process, but special symbols preprocessing wasn't performed during lookup procedure, so that "alzheimer's" didn't contain "alzheimers".