Although current functions behave as intended and desired, add additional functions to handle cesu-8 and modified utf8, as well as guarantee correct disallowing of overlong encodings and invalid sequences (which may already be correct; we just need to prove it). See http://j.mp/1GRqXbX and http://j.mp/1HzJPBY.
Need to provide variant of get_codepoint_from_utf8() that allows customizations of the substitution strategy:
The replacement character "�" (U+FFFD) is what we currently support.
The invalid Unicode code points U+DC80–U+DCFF where the low 8 bits are the byte's value. Sometimes it is called UTF-8B
The Unicode code points U+0080–U+00FF with the same value as the byte, thus interpreting the bytes according to ISO-8859-1
The Unicode code point for the character represented by the byte in CP1252, which is similar to using ISO-8859-1, except that some characters in the range 0x80–0x9F are mapped into different Unicode code points. For example, 0x80 becomes the Euro sign, U+20AC.
Although current functions behave as intended and desired, add additional functions to handle cesu-8 and modified utf8, as well as guarantee correct disallowing of overlong encodings and invalid sequences (which may already be correct; we just need to prove it). See http://j.mp/1GRqXbX and http://j.mp/1HzJPBY.
Need to provide variant of get_codepoint_from_utf8() that allows customizations of the substitution strategy:
May also want to link in libICU.