Buffer over-read in TextDecoder()

[jackkleeman]

TextDecoder decodes a full Buffer when presented with a subarray of a Buffer:

$ llrt -e 'let utf8decoder = new TextDecoder(); console.log(utf8decoder.decode(Buffer.from("abcdefg", "utf8").subarray(0,1)));'
abcdefg
$ llrt -e 'let utf8decoder = new TextDecoder(); console.log(Buffer.from("abcdefg", "utf8").subarray(0,1).toString("utf8"));'
a

[jackkleeman]

Probably related, Buffer.from on a Buffer also overruns

$ llrt -e 'console.log(Buffer.from(Buffer.from("abcdefg", "utf8").subarray(0,1)).toString());'
abcdefg

[richarddavison]

Thanks @jackkleeman for this very clear bug report. We'll be taking a look! Edit: Found the issue. Bytes should not be copied when we create a buffer from a slice. We'll need some slight refactor of the Buffer implementation to correct this.

[jackkleeman]

llrt -e 'const array = new Uint8Array([97, 98, 97, 98]); console.log(new TextDecoder().decode(array.subarray(0,1)))'

The issue also appears to happen with Uint8Array, not just Buffer (maybe same underlying impl?)

[richarddavison]

llrt -e 'const array = new Uint8Array([97, 98, 97, 98]); console.log(new TextDecoder().decode(array.subarray(0,1)))'

The issue also appears to happen with Uint8Array, not just Buffer (maybe same underlying impl?)

Jup, even with strings. Since subarrays shared the same memory, we didn't account for this. This should be fixed in https://github.com/awslabs/llrt/pull/422 (and slightly better performance) since we're no longer copying the buffers around.

[richarddavison]

Fixed in https://github.com/awslabs/llrt/pull/422