Open andydevs opened 3 years ago
Donald Trump's tweets seem to have a lot of Acronyms formatted with dots in between the letters (e.g. "A.O.C"). Because of the period, the Tokenizer splits the acronym into separate words.
...idunno... fix it maybe?
Superseded by #7
andydevs
commented
2 years ago
Donald Trump's tweets seem to have a lot of Acronyms formatted with dots in between the letters (e.g. "A.O.C"). Because of the period, the Tokenizer splits the acronym into separate words.
...idunno... fix it maybe?