Closed greenwoodma closed 5 years ago
If someone tries to use an @mention like a normal word with an apostrophy ending the tokenization gets messed up.
For example, @sadiqkhan't should have a token over the @sadiqkhan portion, but the tokens miss off the last n as that becomes part of a token for n't
@sadiqkhan't
@sadiqkhan
n
n't
If someone tries to use an @mention like a normal word with an apostrophy ending the tokenization gets messed up.
For example,
@sadiqkhan'tshould have a token over the@sadiqkhanportion, but the tokens miss off the lastnas that becomes part of a token forn't