I just caught a very subtle bug which has been in the codebase for 2 years now.
This easily confused variable name caused the effects of the unicode.normalize(raw.text) line above to be ignored 😱
We didn't notice this because it's implemented correctly in sanitizer/_text_pelias_parser.js but incorrectly in sanitizer/_text.js.
You can see that for the /v1/search endpoint it's possible to still send emoji to libpostal:
I suspect that we have been falling back to using the pelias parser more often than needed because in these cases its more likely that libpostal will reject the string containing emoji.
I just caught a very subtle bug which has been in the codebase for 2 years now.
This easily confused variable name caused the effects of the
unicode.normalize(raw.text)line above to be ignored 😱We didn't notice this because it's implemented correctly in
sanitizer/_text_pelias_parser.jsbut incorrectly insanitizer/_text.js.You can see that for the
/v1/searchendpoint it's possible to still send emoji tolibpostal:I suspect that we have been falling back to using the
peliasparser more often than needed because in these cases its more likely thatlibpostalwill reject the string containing emoji.