Closed jahorton closed 1 month ago
Regarding point 1 above - the fact that punctuation is not handled well by auto-correct:
Step 1: Consider the following two possible contexts: apple.
vs apple .
When no whitespace is returned and when the original string indices have been stripped, the prior form of the tokenize()
function made these two contexts indistinguishable. This is a "problem."
Why so?
Step 2: suppose the pre-existing context apple
. Broadly speaking, there are three different cases a follow-up might have:
s
.
(Also consider that there is technically nothing that prohibits a keyboard from specifying a single key that emits a space followed by a '.')
For case 1, we simply continue adding more text to the existing word. No problems here. Predictions are based upon apple
.
For case 2, at present, we explicitly check for a whitespace-only transform. (Whitespaces are the most common case for wordbreaking.) If it occurs, we note this down and prepend it to generated predictions in order to preserve the whitespace. We also generate predictions based on the new empty token following the whitespace.
For case 3... at present, we treat it the same as case 1... despite the fact that wordbreaking will split the context into two pieces: apple
and .
.
apple
and will happily replace the .
while doing so. "."
instead of "apple."
as well!Apple
instead of apple
.It would be best to treat case 3 like case 2. That said, we would still have another issue to resolve to handle this completely ideally. We'd prefer not to 'correct' word-breaking punctuation marks, just as we currently don't 'correct' whitespace when starting a new token after a space.
Toward the issue raised in that last paragraph:
insertAfterWord
appended to predictions, this would make a fantastic check for auto-removing the appended insertAfterWord
.skipCorrection
function to models.
Using the current state of auto-correct this weekend, I ran into a few issues:
Esther
, which is a single letter away fromeither
. The latter is a fairly common word, while the former is a name.