PolygonAPI.getGroupedDaily all or nothing failure

[allenjsomb]

If a single record from grouped daily is bad or unparseable then you get no records at all.

example: if you pull data from 1998-01-20, there is a record that causes a number format exception in the gson deserialization.

wouldn't it be better to return data that was successfully parsed rather than return nothing?

[mainstringargs]

Could you verify what version of the library you're using? Also could you provide the specific API call and parameters you're using? I'm attempting to replicate it on my side.

[allenjsomb]

i'm using 5.0.2. here is a snippet to replicate:

PolygonAPI api = new PolygonAPI(url, wssUrl, polygonKey);
LocalDate date = LocalDate.of(1998, 1, 20);
GroupedDailyResponse resp = api.getGroupedDaily("US", Market.STOCKS, date, false);

this will throw this exception:

com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected an int but was 5.552765625e+09 at line 1 column 2204 path $.results[27].v
    at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:228) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:218) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[gson-2.8.6.jar!/:na]
    at com.google.gson.Gson.fromJson(Gson.java:932) ~[gson-2.8.6.jar!/:na]
    at io.github.mainstringargs.abstracts.rest.AbstractRequest.getResponseObject(AbstractRequest.java:258) ~[alpaca-java-5.0.2.jar!/:na]
    at io.github.mainstringargs.polygon.PolygonAPI.getGroupedDaily(PolygonAPI.java:913) ~[alpaca-java-5.0.2.jar!/:na]
...

[mainstringargs]

Thanks, will take a look. Interestingly enough if you flip the "unadjusted" flag, it looks like it works.

On Tue, Jan 7, 2020 at 9:38 AM Jimmy Allen notifications@github.com wrote:

i'm using 5.0.2. here is a snippet to replicate:

PolygonAPI api = new PolygonAPI(url, wssUrl, polygonKey); LocalDate date = LocalDate.of(1998, 1, 20); GroupedDailyResponse resp = api.getGroupedDaily("US", Market.STOCKS, date, false);

this will throw this exception:

com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected an int but was 5.552765625e+09 at line 1 column 2204 path $.results[27].v at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:228) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:218) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[gson-2.8.6.jar!/:na] at com.google.gson.Gson.fromJson(Gson.java:932) ~[gson-2.8.6.jar!/:na] at io.github.mainstringargs.abstracts.rest.AbstractRequest.getResponseObject(AbstractRequest.java:258) ~[alpaca-java-5.0.2.jar!/:na] at io.github.mainstringargs.polygon.PolygonAPI.getGroupedDaily(PolygonAPI.java:913) ~[alpaca-java-5.0.2.jar!/:na] ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mainstringargs/alpaca-java/issues/38?email_source=notifications&email_token=ABVXM4W4B5BTABEU64YYH23Q4SOY7A5CNFSM4KDHBXNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIJIRAQ#issuecomment-571639938, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVXM4TCRXK6SLFCNQW3HXTQ4SOY7ANCNFSM4KDHBXNA .

[mainstringargs]

Wondering if it is really an issue coming from polygon in the data -- but thinking we shouldn't necessarily completely fail if one piece of data is "corrupt"

On Tue, Jan 7, 2020 at 9:44 AM Mark Cloutier mark.l.cloutier@gmail.com wrote:

Thanks, will take a look. Interestingly enough if you flip the "unadjusted" flag, it looks like it works.

On Tue, Jan 7, 2020 at 9:38 AM Jimmy Allen notifications@github.com wrote:

i'm using 5.0.2. here is a snippet to replicate:

PolygonAPI api = new PolygonAPI(url, wssUrl, polygonKey); LocalDate date = LocalDate.of(1998, 1, 20); GroupedDailyResponse resp = api.getGroupedDaily("US", Market.STOCKS, date, false);

this will throw this exception:

com.google.gson.JsonSyntaxException: java.lang.NumberFormatException: Expected an int but was 5.552765625e+09 at line 1 column 2204 path $.results[27].v at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:228) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.TypeAdapters$7.read(TypeAdapters.java:218) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:41) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:82) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:61) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[gson-2.8.6.jar!/:na] at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[gson-2.8.6.jar!/:na] at com.google.gson.Gson.fromJson(Gson.java:932) ~[gson-2.8.6.jar!/:na] at io.github.mainstringargs.abstracts.rest.AbstractRequest.getResponseObject(AbstractRequest.java:258) ~[alpaca-java-5.0.2.jar!/:na] at io.github.mainstringargs.polygon.PolygonAPI.getGroupedDaily(PolygonAPI.java:913) ~[alpaca-java-5.0.2.jar!/:na] ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/mainstringargs/alpaca-java/issues/38?email_source=notifications&email_token=ABVXM4W4B5BTABEU64YYH23Q4SOY7A5CNFSM4KDHBXNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIJIRAQ#issuecomment-571639938, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABVXM4TCRXK6SLFCNQW3HXTQ4SOY7ANCNFSM4KDHBXNA .

[allenjsomb]

thanks for looking at it. i really like the library and if i am able to get some free time in the future i'd like to help contribute.

[mainstringargs]

Found the underlying problem I think -- when unadjusted is set to true:

api.getGroupedDaily("US", Market.STOCKS, date, true);

This volume in the HEI entry is set to:

{"T":"HEI","v":5.832704e+08,"o":26.63,"c":26.63,"h":26.75,"l":26.63,"t":885330000000}

When unadjusted is set to false:

api.getGroupedDaily("US", Market.STOCKS, date, false);

This volume in the HEI entry is set to:

{"T":"HEI","v":5.552765625e+09,"o":2.7973,"c":2.7973,"h":2.8098,"l":2.7973,"t":885330000000}

5.552765625e+09 is greater than the maximum value for Integer (2,147,483,647). This causes the parsing problem. Updating the schema to use a Long for Aggregrate volume seems to fix the issue. Doing a little bit more testing, and will include it in the next release if my thinking is correct.

[mainstringargs]

I have just updated the library with the tweak, please let me know if you see any other issues. Thank you!

[allenjsomb]

awesome. thx!