Open xei opened 3 years ago
Thanks @xei for reporting this issue. I know BERT has a limit of 512 characters and the model currently being used for inference was trained with maximum 512 characters REF.
Also, I am not sure how corpus_raw
looks like. But 512 character should work for most cases as the spell check only considers a sentence for the context to spell checking and not the entire corpus.
As you can see above the entire text moved through the spacy pipeline without any error. The sample text is taken from The Project Gutenberg eBook of The Junior Trophy, by Ralph Henry Barbour REF.
There is another thing which I wanted to point was contextualSpellCheck would require both parser
and ner
as mentioned here:
We require NER to identify if a token is a PERSON also require parser because we use
Token.sent
for context
Please let me know if you have any questions. I think your suggestion is great, and I will have to try to think of a solution to either split a large sentence (> max_position_embeddings) or bypass spell check altogether. If you would like to contribute this feature feel free to create a PR!
I tried to correct spelling mistakes in a large text.
At first, I faced this error: ValueError: [E030] Sentence boundaries unset. You can add the 'sentencizer' component to the pipeline with:
nlp.add_pipe('sentencizer')
. Alternatively, add the dependency parser or sentence recognizer, or set sentence boundaries by settingdoc[i].is_sent_start
.So, I added the
sentencizer
component to the pipeline.This time I faced this error: RuntimeError: The expanded size of the tensor (837) must match the existing size (512) at non-singleton dimension 1. Target sizes: [1, 837]. Tensor sizes: [1, 512]
I guess this is due to the limitations of BERT. However, I believe that there should be a way to catch this error and bypass the spell check.