Map each code point of an ill-formed UTF-8 subsequence to a replacement character individually

[fzhinkin]

As it was pointed out in https://github.com/Kotlin/kotlinx-io/pull/290#discussion_r1567268068, kotlinx-io converts different ill-formed UTF-8 subsequences differently: either the whole multi-code-point subsequence replaced with a single replacement character, or each code points is converted separately:

• 0xf0 0x89 0x89 <EOF> -> �
• 0xf0 0x89 0x89 0x89 <EOF> -> �
• 0xf0 0xf0 0xf0 <EOF> -> ���

The UTF-8 spec allows handling these ill-formed sequences whatever way we want as long as errors are somehow reported. However, such behavior looks a bit inconsistent and it's hard to reason about how an arbitrary byte sequences will be converted.

We should improve the way ill-formed sequences are handled and stick to an approach adopted by other languages/libraries: convert only ill-formed subsequences consisting of a single byte.

That's how it's done in:

• Java:

  jshell> new String(new byte[]{(byte)0xf0,(byte)0x89,(byte)0x89,(byte)0x89})
  $5 ==> "����"

• Python 3:

  >>> b'\xf0\x89\x89\x89'.decode("utf-8", errors='replace')
  '����

• Go:

  
  fmt.Println(string([]byte{0xf0, 0x89, 0x89, 0x89}))
  ...

����

[ilya-g]

See also the recommendation "U+FFFD Substitution of Maximal Subparts" in https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf

[fzhinkin]

It seems like kotlinx-io behavior could be aligned w/ Kotlin Stdlib (ByteArray.decodeToString in particular) in all scenarios except surrogate code points handling. On JVM, byte-sequences encoding surrogate code points are replaced with a single �, on all other platforms with ���:

ubyteArrayOf(0xedu, 0xbfu, 0xbfu).asByteArray().decodeToString()

https://pl.kotl.in/LMjgmMVGX