polygon-io / client-go

The official Go client library for the Polygon REST and WebSocket API.
MIT License
133 stars 38 forks source link

Improve JSON performance by setting or allowing a compatible parser to be set on resty #353

Closed weirdninja closed 2 months ago

weirdninja commented 1 year ago

The standard json parsing library is reasonably slow in comparison to some other alternatives. The Resty folks describe in https://github.com/go-resty/resty/issues/76#issuecomment-314015250 the few lines needed to change Resty's json parser to something else.

The project could settle on jsoniter or perhaps something else, but in any case, we should be allowed to override this in case something better comes up to be future proof.

justinpolygon commented 10 months ago

Hey @weirdninja, have you written anything here or patched your client for this? Just wondering what your use-case is? I know for the websocket we support raw message types where you can roll your own parser. Maybe we could use something similar for the REST API and just return the raw messages via a flag and then you roll your own parser via whatever library you want.

weirdninja commented 10 months ago

Hi!

I noticed that, for the REST calls, I could set the parser directly on resty since the client is exposed.

So something like:

client := polygon.New("token")
client.Client.HTTP.SetJSONMarshaler(jsoniter.Marshal)
client.Client.HTTP.SetJSONUnmarshaler(jsoniter.Unmarshal)

worked fine for the rest calls.

I have a task to look into a way to implement jsoniter for websockets. I will eventually get to it. Are there any pointers you could provide?

weirdninja commented 10 months ago

About the use case: a pperf analysis of our client usage showed a significant amount of time parsing json due to the size of the option chains. This implementation did boost things by some 10-20%. I don't have the numbers handy, but it was significant.