DataStream.slice throws 'write after end' error

[maxharlow]

With this file, test.ndjson:

{ "abc": 1 }
{ "abc": 2 }
{ "abc": 3 }
{ "abc": 4 }

And this code:

const FS = require('fs')
const Scramjet = require('scramjet')

Scramjet.StringStream.from(FS.createReadStream('test.ndjson')).JSONParse().slice(0, 1).toArray()

Throws this error:

UnhandledPromiseRejectionWarning: Error [ERR_STREAM_WRITE_AFTER_END]: write after end
    at DataStream.Writable.write (_stream_writable.js:292:11)
    at StringStream.ondata (_stream_readable.js:719:22)
    at StringStream.emit (events.js:315:20)
    at StringStream.EventEmitter.emit (domain.js:548:15)
    at addChunk (_stream_readable.js:309:12)
    at readableAddChunk (_stream_readable.js:284:9)
    at StringStream.Readable.push (_stream_readable.js:223:10)
    at StringStream.Transform.push (_stream_transform.js:166:32)
    at /Users/maxharlow/Projects/test/node_modules/scramjet-core/lib/util/mk-transform.js:71:38
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

Bizarrely, if you remove one line from the file (any line), you don't get the error.

Node v14.14, Scramjet v4.35.6

[MichalCz]

Hey,

Thanks for the info - can you downgrade to 4.34 and 4.35.5 and see if any of these has an effect.

I'll add the above case to tests but it will take a bit to reproduce, so you may solve the issue this way.

M.

[maxharlow]

Thanks for the quick response! I still get the same error on v4.34, but I tried going back to v4.19 and that seems to work

[MichalCz]

Ok. That's a good lead - hopefully we haven't added some specific function in between 4.19 and now. I can't remember any prod-grade vulnerable deps either so if that works for you, as a dependency it is safe to use it for now.

I think I know what is the reason so it shouldn't be long until I manage to fix it, rather days than weeks.

[MichalCz]

It turned out to be hours... I've got a fix - release tomorrow.

[maxharlow]

Excellent news, thank you!

[MichalCz]

@maxharlow - check scramjet@4.35.7...

...and your LinkedIn BTW ;)