pamelafox / ndjson-readablestream

A small JS package for reading a ReadableStream of NDJSON
MIT License
9 stars 4 forks source link

Malformed JSON can break streaming #11

Open IanVS opened 4 months ago

IanVS commented 4 months ago

First off, thanks for this package and the associated blog post, they've been very helpful. I've been experimenting with it, and found that if one line has malformed JSON (missing an end brace, for instance), streaming stops for all subsequent lines as well.

I see this is because although the response stream is split on newlines, the try/catch is not done per-line, but rather on the running concatenation of all of the text thus far. Can I ask, what is the reason for not trusting newlines, and only parsing each line within a try/catch? This way, one malformed object could be rejected (and maybe an onError callback could be triggered), while continuing to process subsequent lines? I'm sure there's something I'm missing, but if that could work, I'd be happy to submit a PR if you're interested.

Thanks again!

pamelafox commented 4 months ago

Interesting, I didn't have a situation with malformed JSON, so I didn't dig into it. I'm curious if the related specifications for ndjson/jsonlines/json-seq talk about malformed lines and how to handle them? The closest I find is here: https://github.com/ndjson/ndjson-spec?tab=readme-ov-file#32-parsing But that doesn't clarify whether a parser should continue on to the next event. I'm thinking it could be good as an option, like continueAfterError, but default to false?