search

dunglas / vulcain

🔨 Fast and idiomatic client-driven REST APIs.
https://vulcain.rocks
GNU Affero General Public License v3.0
3.51k stars 106 forks source link

Improve memory usage by using a streaming JSON parser and encoder #61

Open dunglas opened 3 years ago

dunglas commented 3 years ago

Currently, we have to wait for the response from the upstream server to be entirely sent, the to load it entirely in memory to analyze and modify the received JSON document. Analyzing and modifying the JSON stream on the go could improve the memory usage (but probably not the overall performance because we need to compute and to send the Content-Length HTTP header before the JSON body anyway).

This isn't an easy pick because gjson doesn't support JSON streams for now (https://github.com/tidwall/gjson/issues/25, https://github.com/tidwall/gjson/issues/78). We'll have to use json.Decoder directly or to create a new library for this.

mitar commented 1 month ago

Maybe https://github.com/minio/simdjson-go or https://github.com/ohler55/ojg could be used instead.