nodejs / node

Node.js JavaScript runtime ✨🐢🚀✨
https://nodejs.org
Other
106.58k stars 29.06k forks source link

Missing lines when using readline Interface as AsyncIterable #28565

Open amcheshire opened 5 years ago

amcheshire commented 5 years ago

We were using readline to convert a stream that gives bytes to an AsyncIterable of lines. We did this by using something like:

const iterableOfLines = readline.createInterface({ input: streamOfBytes })

We would then use this as an AsyncIterable (which looks like it should be supported) and write it to a file, but there would sometimes be lines missing. The number of lines missing was inconsistent and we couldn't identify any patterns relating the lines that were missing. They were not different from the rest in any way, they were not always the first or last lines, etc.

Unfortunately, it's hard for me to tell you more because we were unable to write an isolated test that would reliably show this behavior (we noticed it in production).

We are confident that the bug is in readline because we replaced the use of it with our own custom function that would convert the bytes to lines, and we consistently get all of the lines every time.

jasnell commented 4 years ago

@nodejs/stream @BridgeAR ... any ideas?

jfriend00 commented 4 years ago

Probably related to https://github.com/nodejs/node/issues/33463.