Implement Unicode normalisation, to ensure that canonically equivalent strings have an identical representation.
The preprocessing step in the Indic module implements parts of this as required, which should be removed once normalisation has been properly implemented.
We should also investigate the feasibility of using open-source options, e.g. unicode-normalization.
Implement Unicode normalisation, to ensure that canonically equivalent strings have an identical representation.
The preprocessing step in the Indic module implements parts of this as required, which should be removed once normalisation has been properly implemented.
We should also investigate the feasibility of using open-source options, e.g.
unicode-normalization.