polygon-io / issues

Quickly track and report problems with polygon.io
29 stars 0 forks source link

Volume field (integer) output as exponential format - fails in GOLang parser. #172

Closed avikdev closed 1 year ago

avikdev commented 2 years ago

URL https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/minute/2021-11-22/2021-11-22?adjusted=false&sort=asc&limit=5000&apiKey=

Result The volume field in aggregate bars (JSON integer field) are output in exponential format, which makes it difficult to parse in GOLang.

Expected Result Please output all integer fields in decimal format. More details in the additional context below.

Screenshots n/a

Desktop (please complete the following information): n/a

Additional context

The following are a selected few of the broken down entries from the JSON output which correspond to 1-minute bars of the stock "TSLA" on Nov-22. See the 4-th row, the volume field is output in exponential format ("2.384421e+06").

{"v":19218,"vw":161.7487,"a":160.838,"o":161.78,"c":161.7099,"h":161.82,"l":161.62,"t":1637591220000,"n":272}, {"v":3459,"vw":161.6259,"a":160.8386,"o":161.62,"c":161.61,"h":161.66,"l":161.6,"t":1637591280000,"n":170}, {"v":12004,"vw":161.6588,"a":160.8406,"o":161.66,"c":161.65,"h":161.71,"l":161.64,"t":1637591340000,"n":259}, {"v":2.384421e+06,"vw":161.8688,"a":161.1786,"o":161.68,"c":162.45,"h":162.5,"l":161.63,"t":1637591400000,"n":15643,"op":161.68}, {"v":826300,"vw":162.4534,"a":161.3086,"o":162.43,"c":162.55,"h":162.71,"l":162.07,"t":1637591460000,"n":8964,"op":161.68}, {"v":849916,"vw":162.8506,"a":161.4549,"o":162.57,"c":162.9398,"h":163.03,"l":162.5,"t":1637591520000,"n":9706,"op":161.68}, {"v":642363,"vw":162.9553,"a":161.5554,"o":162.93,"c":162.82,"h":163.06,"l":162.81,"t":1637591580000,"n":13189,"op":161.68}, {"v":776326,"vw":162.5225,"a":161.6277,"o":162.81,"c":162.7601,"h":162.81,"l":162.275,"t":1637591640000,"n":6185,"op":161.68},

I am parsing the result in GoLang using the standard "encoding/json" package, which fails to parse the JSON due to this exponential field, I have defined the type as int64 and the actual value is much less than what int64 can hold. The actual error is

"panic: json: cannot unmarshal number 3.331533e+06 into Go struct field _aggregateBar.results.v of type int64"

I know there is a temporary workaround - to use float64 instead of int64, but since the volume is originally an int field, I don't want to do that. And I guess other languages might possibly have this error as well. Is it possible to make the change necessary Polygon API to emit the volume fields always as decimal int64 ?

Thanks.

SwiftlyDeft commented 2 years ago

@avikdev - the format is Javascript's representation of a large integer (could be other languages as well). What we did was store the value in an interface then convert it into an Int. It's just how Golang parses the value. I can see why they did this because it saves space in the response.

Just as a sample we did the following:

    VolumeAtCloseRAW           interface{} `json:"v" bson:", truncate"` 
    VolumeAtClose              int64       `json:"-"`

Then on the struct we created a function like this: func (p *PreviousClose) CopyRawVolume() { if v, ok := p.VolumeAtCloseRAW.(int); ok { p.VolumeAtClose = int64(v) } if v, ok := p.VolumeAtCloseRAW.(int32); ok { p.VolumeAtClose = int64(v) } if v, ok := p.VolumeAtCloseRAW.(float64); ok { p.VolumeAtClose = int64(v) } if v, ok := p.VolumeAtCloseRAW.(int64); ok { p.VolumeAtClose = v } }

It's just an added step and you can make that function more generic so you can use it on other structs.

jrbell19 commented 1 year ago

We recommend using our Go client library: https://github.com/polygon-io/client-go