I just caught a very subtle bug which has been in the codebase for 2 years now.
This easily confused variable name caused the effects of the unicode.normalize(raw.text) line above to be ignored 😱
We didn't notice this because it's implemented correctly in sanitizer/_text_pelias_parser.js but incorrectly in sanitizer/_text.js.
You can see that for the /v1/search endpoint it's possible to still send emoji to libpostal:
I suspect that we have been falling back to using the pelias parser more often than needed because in these cases its more likely that libpostal will reject the string containing emoji.
I just caught a very subtle bug which has been in the codebase for 2 years now.
This easily confused variable name caused the effects of the
unicode.normalize(raw.text)
line above to be ignored 😱We didn't notice this because it's implemented correctly in
sanitizer/_text_pelias_parser.js
but incorrectly insanitizer/_text.js
.You can see that for the
/v1/search
endpoint it's possible to still send emoji tolibpostal
:I suspect that we have been falling back to using the
pelias
parser more often than needed because in these cases its more likely thatlibpostal
will reject the string containing emoji.