Remove U+2063 (invisible separator) from sentence

common-voice / sentence-collector

Tool to collect and review sentences for Common Voice

Mozilla Public License 2.0

81 stars 62 forks source link

Closed bact closed 3 years ago

bact commented 3 years ago

Cleanup: Remove U+2063 (invisible separator) which occurs in Thai text cut & pasted from some text editors (like MS Word and iOS Notes) - to reduce duplicated texts
Validation: STRUCTURE_REGEX to prevent having “” ‘’ and ` in a long running text (of 55 or more characters)

Note: in the Sentence Extractor, the plan is to remove U+2063 as well (same as U+200b and U+200c zero-width chars)

MichaelKohler commented 3 years ago

:tada: This PR is included in version 2.3.8 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: