`IO#gets(limit)` could produce incomplete UTF-8 strings

[HertzDevil]

If the limit argument is passed to IO#gets, it behaves like a hard limit and the IO always consumes that number of bytes when the delimiter is not found. This means it may stop in the middle of a UTF-8 byte sequence:

str = "ＡＢＣＤ" # => "\xEF\xBC\xA1\xEF\xBC\xA2\xEF\xBC\xA3\xEF\xBC\xA4"
(0..16).each do |i|
  p [i, IO::Memory.new(str).gets(i)]
end

[0, ""]
[1, "\xEF"]
[2, "\xEF\xBC"]
[3, "Ａ"]
[4, "Ａ\xEF"]
[5, "Ａ\xEF\xBC"]
[6, "ＡＢ"]
[7, "ＡＢ\xEF"]
[8, "ＡＢ\xEF\xBC"]
[9, "ＡＢＣ"]
[10, "ＡＢＣ\xEF"]
[11, "ＡＢＣ\xEF\xBC"]
[12, "ＡＢＣＤ"]
[13, "ＡＢＣＤ"]
[14, "ＡＢＣＤ"]
[15, "ＡＢＣＤ"]
[16, "ＡＢＣＤ"]

It also seems to ignore custom encodings, so limit represents the maximum number of bytes returned, not the maximum number of bytes read:

str = "ＡＢＣＤ".encode("UTF-32LE")
(0..16).each do |i|
  io = IO::Memory.new(str)
  io.set_encoding "UTF-32LE"
  p [i, io.gets(i)]
  # same results as above, even though each character now occupies 4 bytes instead of 3
end

Ruby's behavior is to consume additional bytes until a character in the IO's encoding is fully consumed, whether the IO uses UTF-8 or not:

require 'stringio'

str = "ＡＢＣＤ"
(0..16).each do |i|
  p [i, StringIO.new(str).gets(i)]
end

str = "ＡＢＣＤ".encode(Encoding::UTF_32LE)
(0..16).each do |i|
  # here the `StringIO`'s external encoding is automatically set to UTF-32LE
  p [i, StringIO.new(str).gets(i).encode(Encoding::UTF_8)]
end

[0, ""]
[1, "Ａ"]
[2, "Ａ"]
[3, "Ａ"]
[4, "ＡＢ"]
[5, "ＡＢ"]
[6, "ＡＢ"]
[7, "ＡＢＣ"]
[8, "ＡＢＣ"]
[9, "ＡＢＣ"]
[10, "ＡＢＣＤ"]
[11, "ＡＢＣＤ"]
[12, "ＡＢＣＤ"]
[13, "ＡＢＣＤ"]
[14, "ＡＢＣＤ"]
[15, "ＡＢＣＤ"]
[16, "ＡＢＣＤ"]
[0, ""]
[1, "Ａ"]
[2, "Ａ"]
[3, "Ａ"]
[4, "Ａ"]
[5, "ＡＢ"]
[6, "ＡＢ"]
[7, "ＡＢ"]
[8, "ＡＢ"]
[9, "ＡＢＣ"]
[10, "ＡＢＣ"]
[11, "ＡＢＣ"]
[12, "ＡＢＣ"]
[13, "ＡＢＣＤ"]
[14, "ＡＢＣＤ"]
[15, "ＡＢＣＤ"]
[16, "ＡＢＣＤ"]

It does so for successive #gets calls too:

io = StringIO.new "ＡＢＣＤ"
io.gets(7) # => "ＡＢＣ"
io.gets(1) # => "Ｄ"

io = StringIO.new "ＡＢＣＤ".encode(Encoding::UTF_32LE)
io.gets(7).encode(Encoding::UTF_8) # => "ＡＢ"
io.gets(1).encode(Encoding::UTF_8) # => "Ｃ"

Contrast this with Crystal:

io = IO::Memory.new "ＡＢＣＤ"
io.gets(7) # => "ＡＢ\xEF"
io.gets(1) # => "\xBC"

io = IO::Memory.new "ＡＢＣＤ".encode("UTF-32LE")
io.set_encoding "UTF-32LE"
io.gets(7) # => "ＡＢ\xEF"
io.gets(1) # => "\xBC"

I think we should follow Ruby's behavior here. Note that we cannot consume less than limit bytes here; not all IOs are seekable, and we don't have something like #unget, so we cannot in general discard bytes forming an incomplete sequence after they are read.

[HertzDevil]

Is it even a good idea to leave it a byte limit? Could we have overloads with character limits here?

[asterite]

Oh, I think this is I bug I probably introduced when we added IO#peek and the related optimization in IO#gets (gets_peek). If you remove that, and also remove the overwritten IO::Memory#gets method, it works fine.

My thoughts: we should exactly like Ruby here. In Ruby limit is bytes, but if you are in the middle of a codepoint you just continue until the end of it. You end up returning more bytes than requested, but a limit is always a soft-limit in my mind.