Investigate potential NFC / NFD Unicode normalization issues for composed characters (accents etc.)

openfoodfacts / openfoodfacts-server

Open Food Facts database, API server and web interface - 🐪🦋 Perl, CSS and JS coders welcome 😊 For helping in Python, see Robotoff or taxonomy-editor

http://openfoodfacts.github.io/openfoodfacts-server/

GNU Affero General Public License v3.0

658 stars 387 forks source link

Investigate potential NFC / NFD Unicode normalization issues for composed characters (accents etc.) #8225

Open stephanegigandet opened 1 year ago

stephanegigandet commented 1 year ago

We currently do not normalize Unicode strings, so it is very likely that some string comparisons (e.g. to match strings to taxonomies) do not work if the strings and the taxonomy entries have different normalization.

This issue is to:

verify if we do have issues
add test cases for those issues
fix those issues

It may be a good thing to normalize all strings in OFF (taxonomies, database, API etc.) to a specific form.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity.