search

phiresky / pandoc-url2cite

Effortlessly and transparently add correctly styled citations to your markdown paper given only a URL
Other
122 stars 9 forks source link

JSON string is truncated when read from stdin #3

Closed LumaLivy closed 4 years ago

LumaLivy commented 4 years ago

Hello, I'll try to provide as many details as I can.

I ran into a rather interesting problem when trying to compile my PDF. Whether it's through some combination of commands and Atom IDE packages, I was getting an "unexpected end of JSON input" error here: https://github.com/phiresky/pandoc-url2cite/blob/7ab6c99cb5833384cac60284aff22b68c4c648b8/pandoc-url2cite.ts#L16

This was happening as soon as my markdown file reached a certain amount of characters, so I went into the code and manually started trying to log what kind of string JSON.parse was seeing. Sure enough, the JSON string was being truncated, causing JSON.parse to throw the error. After scratching my head for a bit and researching around online, I ended up trying an alternate method of reading from stdin that I found on StackOverflow here which ended up working:

I added this at the top of the file:

async function read(stream) {
    const chunks = [];
    for await (const chunk of stream) chunks.push(chunk);
    return Buffer.concat(chunks).toString("utf8");
}

and changed the line I linked above to this:

let data = JSON.parse(await read(process.stdin));
// let data = JSON.parse(fs.readFileSync(0, "utf-8"));

The result is a properly parsed object and no more errors. There seemed to be no discernible difference in speed on a ~1000 word file.

Rather than make a pull request in my 4am stupor I decided to make this an issue since this feels a bit like a hack and I haven't taken a wide look at the repo yet. I thought it might be a good idea for you to have a look and maybe implement this properly from your end since you have more experience. I was thinking it might also be an issue specific to my setup, I'm not sure. I'd be interested in hearing your thoughts though, given that it seems like a simple enough alternative. I wouldn't mind taking a deeper look and making the pull request sometime in the near future if you give me the green light.

Some other info: I'm on Windows 10, was running this from Atom using a package that allows compilation of a pandoc/markdown file on save. Worked like a charm right up until I reached a certain amount of characters.

By the way, thank you for the package. It's making my life a lot easier for referencing/citations in my report!

phiresky commented 4 years ago

Thank you for your report.

I don't really understand why readFileSync might have truncated output and other methods not. The only thing I can think of is that stdin/stdout handling on Windows is pretty weird in general. Other Windows users seem to have an issue with this too, at least in the past.

The readFileSync was kind of a hack in the first place, so I've replaced the reading implementation with the one you suggested and published a new version to npm.