Reconstruct original text without extra whitespace tokens after highlighting

castorini / covidex

A multi-stage neural search engine for the COVID-19 Open Research Dataset

https://covidex.ai

MIT License

137 stars 27 forks source link

Closed edwinzhng closed 4 years ago

edwinzhng commented 4 years ago

Extra whitespace is added to text after tokenization and decoding from highlighting, we can remove the extra whitespace using the original text and update highlighted text characters accordingly

lintool commented 4 years ago

In particular, "COVID - 19", "SARS - CoV - 2", "2019 - nCov" come up repeatedly.