avian2 / unidecode

ASCII transliterations of Unicode text - GitHub mirror
https://pypi.python.org/pypi/Unidecode
GNU General Public License v2.0
516 stars 62 forks source link

Feature Request: Add ability to set custom replacements #54

Open atlowell-smpl opened 4 years ago

atlowell-smpl commented 4 years ago

This is a feature request to add a library function that can be called to set custom character replacement mappings. The library function could be called in a similar manner to as follows:

unidecode.setReplacement('™', "(TM)")

after which subsequent calls to unidecode would replace '™' with '(TM)' rather than the default of "(tm).

For my particular use case, this would be useful for standardizing names across data sources. For instance, if one source has the name "Acme™" and the other has the name "Acme", I could make a call to replace all instances of '™' with "", which allows me to easily compare across sources.

However, I could see a large number of other use cases for this as well. For example, in the FAQ, it is mentioned that localization is not supported and that certain german characters translate to english text rather than german phonetics. Having a call like this could allow programmers to make their own "localization" libraries, allowing them to whatever mappings they feel like.

youssefavx commented 4 years ago

I would love this. I'd love to be able to replace something like: [?] with an empty space instead.

avian2 commented 4 years ago

I think @atlowell-smpl's use case is something that falls outside of Unidecode's concern. It's easy to do such string replacements yourself before a call to Unidecode.

I agree that the characters that get replaced with a [?] are a problem. It's something that #53 already tried to address (reading my comments there I think I initially misunderstood the goal of that pull request).

Before code for that can be added the character tables first need to be updated and unknown characters marked (probably with None instead of [?])

avian2 commented 3 years ago

I've implemented the feature to specify custom replacement strings for characters that are unknown to Unidecode. For example, this will use ASCII space to replace characters that are not are not present in Unidecode's tables.

>>> unidecode('[\ue000]', errors='replace', replace_str=' ')
[ ]