Improve handling of texts with misaligned tokens

explosion / spacy-stanza

💥 Use the latest Stanza (StanfordNLP) research models directly in spaCy

MIT License

722 stars 60 forks source link

Closed adrianeboyd closed 4 years ago

adrianeboyd commented 4 years ago

Add a warning when tokens can't be aligned with the text due to MWT expansion
Skip offset-based checks when processing the stanza tokens for texts with MWT expansion
Add a warning when the offset-based checks detect problems for texts that should be aligned

adrianeboyd commented 4 years ago

I'm never sure how best to word all the warnings, suggestions/changes are welcome.