Closed Holek closed 8 years ago
Hmpf. This is legit. Let me see if I can find some time this week to work on this -- I think Rinku is due for some UTF8 handling overhaul.
Rinku is generating invalid UTF-8 strings for us as well. We can give it valid_encoding?
utf-8 inputs and receive invalid outputs.
Here's a reproduction case:
content = "<a href='http://gi.co'>gi.co</a> r" # the last whitespace is a char 160 non-breaking space, don't copy from GitHub formatted output, click 'edit'
p content[-2].codepoints
# => [160]
p content.valid_encoding?
# => true
p Rinku.auto_link(content).valid_encoding?
# => false
Say I have a Unicode link:
https://pl.wikipedia.org/wiki/Komisja_śledcza_do_zbadania_sprawy_zarzutu_nielegalnego_wywierania_wpływu_na_funkcjonariuszy_policji,_służb_specjalnych,_prokuratorów_i_osoby_pełniące_funkcje_w_organach_wymiaru_sprawiedliwości
Characters I get for this string is:
"ą" character here is split in two bytes:
["\xC4", "\x85"]
.\x85
is a new-line character in UTF-8, so I assume that it's interpreted as such, rather than a part of multi-byte sequence.Maybe #39 is connected?