search

michalmuskala / jason

A blazing fast JSON parser and generator in pure Elixir.
Other
1.61k stars 169 forks source link

Jason.decode timeouts silently with large files #154

Closed herman-rogers closed 1 year ago

herman-rogers commented 2 years ago

So we're processing lots of data and we pull back a record with 300k lines in it. I'd expect Jason parse to just fail or error out but what happens is nothing - it returns and our system continues forward. I tried both Jason.decode! and Jason.decode with the same result. It just stops processing and the program continues. Ideally we'd have an error throw up because of this so we can manage the size.

michalmuskala commented 2 years ago

This sounds quite suspicious to me. In particular, there's no place inside of Jason where there are any timeouts.

Do you have a way for me to reproduce the issue? I don't think we'll be able to progress much without that

herman-rogers commented 2 years ago

I can provide a JSON file that can reproduce this issue although I will say I don't believe it is timing out. I ran it for a couple of hours and it eventually parsed the entire file - it just took a very long time.

Edit: So this can probably be closed - it is a file with 200k lines so it could just be that we need to feed it smaller files for processing.

herman-rogers commented 2 years ago

Actually there might be a performance issue - with Jason.decode a relatively smallish file takes 3 minutes to parse while Poison.decode takes ~5 to 10 seconds

michalmuskala commented 2 years ago

I'd love to get my hands on this file. That indeed sounds like a bug.

You can send it either here as attachment or to my email michal@muskala.eu

michalmuskala commented 1 year ago

Given no progress here, I'm going to close. If you can provide more details, please re-open.