rewicks / ersatz

Apache License 2.0
39 stars 5 forks source link

Removing spaces when splitting sentences #10

Open fatihbeyhan opened 1 year ago

fatihbeyhan commented 1 year ago

Hi!

Thank you for your work! I am trying to use your multilingual model for Turkish text. The model is not restoring the spaces. I am trying your model on Annotated data, therefore every character is essential to preserve the annotated char sequences. When there is a paragraph with some sentences using two spaces between words instead of one, the split version of these sentences removes these multi-spaces. Any suggestions?