starwing / luautf8

a utf-8 support module for Lua and LuaJIT.
MIT License
406 stars 67 forks source link

Invalid byte sequence treating #27

Closed johnd0e closed 4 years ago

johnd0e commented 4 years ago

I found that luautf8 in some cases treats some invalid sequences as correct utf8. For example, utf8 string cannot consist of single char in range 128..255. So when we execute following code in lua 5.3 we get no output:

for i=128,255 do
  local c = string.char(i)
  if utf8.len(c) then
    print(i)
  end
end

But luautf8 considers all such strings as valid.

Even more: luautf8 treats some invalid chars as punctuation:

for i=128,255 do
  local c = string.char(i)
  print(
    i,
    utf8.match(c,'%p'),
    string.match(c,'%p')
  )
end
johnd0e commented 4 years ago

Perhaps, this is duplicate of #13