search

mailru / easyjson

Fast JSON serializer for golang.
MIT License
4.48k stars 421 forks source link

jlexer: Support io.Reader and read in chunks #287

Open mrVanboy opened 4 years ago

mrVanboy commented 4 years ago

Hi! Thanks for a great library, we are using it a lot in our projects.

What I found today interesting in our profiler, that easyjson.UnmarshalFromReader reads everything from the io.Reader before further processing, which causes huge allocations in case of large data stored in the provided Reader. Yep, it's mentioned in the documentation of the easyjson.UnmarhsalFromReader, but I see here space for improvement.

Profiler screenshot ![image](https://user-images.githubusercontent.com/19638942/80355869-e8b74780-8878-11ea-91e6-37b64a146562.png)

Have you considered reading and parsing chunks of the bytes from io.Reader?

It could help with parsing huge data from net/http.Response.Body as in documentation has mentioned:

The response body is streamed on demand as the Body field is read...

shmel1k commented 4 years ago

Hi! Yes, we did some kind of attempts and found it a little bit difficult to implement immediately :(

mdaliyan commented 4 years ago

I totally agree, all my allocations are coming from ioutil.ReadAll that I need to call before sending data to easyJson.

I really got happy when I saw UnmarshalFromReader function and but my joy disappeared when I saw its code that uses ioutil.ReadAll too.