Duplicate stream trade data

[WinJ]

Hi,

I'm using WebSocketClient to consume stream trade data and I noticed for some stocks the trade data have duplicates while some stocks are OK. For example, below is one stream message for PAYC today: [{“ev”:“T”,“sym”:“PAYC”,“i”:"52983527388738 ",“x”:11,“p”:281.1,“s”:2,“c”:[14,37,41],“t”:1596206704765,“z”:1},{“ev”:“T”,“sym”:“PAYC”,"i ":“52983527388738”,“x”:11,“p”:281.1,“s”:2,“c”:[14,37,41],“t”:1596206704765,“z”:1}]

For this data, it returns two trades in one message, with exact same content including trade ID (annotated by “i”). I assume trade ID is unique, is that right? There are some other examples that same two trades are in separate messages.

Do we know why there are duplicate trade data in the message? Shall we treat them as one trade data? I checked Polygon agg bar API, and apparently the API treat them as one trade.

Really appreciate the help!

[jrbell19]

We are currently looking into this. If you see any other tickers that are sending duplicates please let me know.

Apologies for the inconvenience

[WinJ]

I do noticed more tickers that have this issue, for example, BDX, CIEN. But I believe that's not an exhausted list.

Also want to confirm, we should be use trade ID as unique identifier, right?

[WinJ]

To add a few more tickers with the issue: PINS, DIS, NOW, CLX, SAM

Hope it helps with debug.

[FreeGoldRush]

Can someone please confirm that the trade ID is unique and can be used to filter out these duplicates? But that brings up the obvious: If it is unique then why isn't polygon filtering out duplicates?

[sasha-id]

Looks like all symbols are affected today, here's sample dump:

I, [2020-09-01T15:29:39.914723 #37834]  INFO -- : [{"ev":"T","sym":"TDOC","i":"52983575561794","x":10,"p":222.39,"s":27,"c":[14,37,41],"t":1598988579874,"z":1},{"ev":"T","sym":"TDOC","i":"52983575561794","x":10,"p":222.39,"s":27,"c":[14,37,41],"t":1598988579874,"z":1},{"ev":"T","sym":"TDOC","i":"52983575561795","x":10,"p":222.39,"s":23,"c":[37],"t":1598988579874,"z":1},{"ev":"T","sym":"TDOC","i":"52983575561795","x":10,"p":222.39,"s":23,"c":[37],"t":1598988579874,"z":1}]
I, [2020-09-01T15:29:39.914867 #37834]  INFO -- : [{"ev":"T","sym":"DAO","i":"52983558788689","x":10,"p":31.96,"s":100,"c":[14,41],"t":1598988579832,"z":1},{"ev":"T","sym":"DAO","i":"52983558788689","x":10,"p":31.96,"s":100,"c":[14,41],"t":1598988579832,"z":1},{"ev":"T","sym":"DAO","i":"79372493076305","x":4,"p":31.9582,"s":100,"t":1598988579835,"z":1},{"ev":"T","sym":"DAO","i":"79372493076305","x":4,"p":31.9582,"s":100,"t":1598988579835,"z":1}]
I, [2020-09-01T15:29:39.915046 #37834]  INFO -- : [{"ev":"T","sym":"EDUC","i":"135","x":4,"p":16.782,"s":300,"t":1598988579864,"z":3}]
I, [2020-09-01T15:29:39.919248 #37834]  INFO -- : [{"ev":"T","sym":"BNTX","i":"10728","x":12,"p":56.9,"s":20,"c":[37],"t":1598988579895,"z":3}]
I, [2020-09-01T15:29:39.919380 #37834]  INFO -- : [{"ev":"T","sym":"PRAH","i":"603","x":3,"p":105.91,"s":8,"c":[14,37,41],"t":1598988579831,"z":3}]
I, [2020-09-01T15:29:39.932296 #37834]  INFO -- : [{"ev":"T","sym":"DOCU","i":"35426","x":4,"p":266.99,"s":50,"c":[37],"t":1598988579867,"z":3},{"ev":"T","sym":"DOCU","i":"78183","x":4,"p":266.99,"s":170,"t":1598988579891,"z":3},{"ev":"T","sym":"BLFS","i":"1049","x":12,"p":22.59,"s":21,"c":[14,37,41],"t":1598988579895,"z":3},{"ev":"T","sym":"BLFS","i":"1050","x":12,"p":22.59,"s":76,"c":[14,37,41],"t":1598988579895,"z":3}]
I, [2020-09-01T15:29:39.932404 #37834]  INFO -- : [{"ev":"T","sym":"INGN","i":"462","x":4,"p":28.79,"s":32,"c":[37],"t":1598988579913,"z":3},{"ev":"T","sym":"GNRC","i":"62879871164722","x":12,"p":193.34,"s":5,"c":[14,37,41],"t":1598988579873,"z":1},{"ev":"T","sym":"GNRC","i":"62879871164722","x":12,"p":193.34,"s":5,"c":[14,37,41],"t":1598988579873,"z":1}]
I, [2020-09-01T15:29:39.932709 #37834]  INFO -- : [{"ev":"T","sym":"MRCY","i":"1691","x":12,"p":76.16,"s":2,"c":[37],"t":1598988579882,"z":3},{"ev":"T","sym":"MRCY","i":"1692","x":12,"p":76.16,"s":1,"c":[37],"t":1598988579882,"z":3}]
I, [2020-09-01T15:29:39.933016 #37834]  INFO -- : [{"ev":"T","sym":"TWLO","i":"79372493076306","x":4,"p":272.3344,"s":2,"c":[37],"t":1598988579864,"z":1},{"ev":"T","sym":"OFC","i":"71688443282007","x":4,"p":24.31,"s":100,"t":1598988579874,"z":1},{"ev":"T","sym":"TWLO","i":"79372493076306","x":4,"p":272.3344,"s":2,"c":[37],"t":1598988579864,"z":1},{"ev":"T","sym":"OFC","i":"71688443282007","x":4,"p":24.31,"s":100,"t":1598988579874,"z":1}]
I, [2020-09-01T15:29:39.939535 #37834]  INFO -- : [{"ev":"T","sym":"GOCO","i":"1679","x":12,"p":13.57,"s":169,"c":[14,41],"t":1598988579887,"z":3},{"ev":"T","sym":"GOCO","i":"297","x":19,"p":13.57,"s":2,"c":[37],"t":1598988579887,"z":3},{"ev":"T","sym":"GOCO","i":"1680","x":12,"p":13.57,"s":100,"c":[14,41],"t":1598988579887,"z":3},{"ev":"T","sym":"GOCO","i":"699","x":11,"p":13.57,"s":100,"c":[14,41],"t":1598988579887,"z":3},{"ev":"T","sym":"GOCO","i":"155","x":10,"p":13.56,"s":100,"c":[14,41],"t":1598988579887,"z":3},{"ev":"T","sym":"GOCO","i":"156","x":10,"p":13.56,"s":100,"t":1598988579887,"z":3},{"ev":"T","sym":"GOCO","i":"350","x":15,"p":13.55,"s":100,"t":1598988579894,"z":3}]
I, [2020-09-01T15:29:39.939654 #37834]  INFO -- : [{"ev":"T","sym":"TTOO","i":"11225","x":4,"p":1.6099,"s":1,"c":[37],"t":1598988579857,"z":3},{"ev":"T","sym":"TTOO","i":"11226","x":4,"p":1.61,"s":1,"c":[37],"t":1598988579857,"z":3},{"ev":"T","sym":"TTOO","i":"11227","x":4,"p":1.6099,"s":8,"c":[37],"t":1598988579857,"z":3},{"ev":"T","sym":"TTOO","i":"8445","x":4,"p":1.61,"s":8,"c":[37],"t":1598988579858,"z":3}]
I, [2020-09-01T15:29:39.939897 #37834]  INFO -- : [{"ev":"T","sym":"CHWY","i":"71688443282008","x":4,"p":69.4399,"s":2,"c":[37],"t":1598988579881,"z":1},{"ev":"T","sym":"MCO","i":"62879165463868","x":2,"p":294.94,"s":1,"c":[37],"t":1598988579903,"z":1},{"ev":"T","sym":"MCO","i":"62879165463868","x":2,"p":294.94,"s":1,"c":[37],"t":1598988579903,"z":1}]
I, [2020-09-01T15:29:39.945292 #37834]  INFO -- : [{"ev":"T","sym":"GEVO","i":"11498","x":4,"p":1.2399,"s":16,"c":[37],"t":1598988579924,"z":3},{"ev":"T","sym":"GEVO","i":"9306","x":4,"p":1.24,"s":16,"c":[37],"t":1598988579924,"z":3}]
I, [2020-09-01T15:29:39.945406 #37834]  INFO -- : [{"ev":"T","sym":"FMBI","i":"340","x":4,"p":12.26,"s":85,"c":[37],"t":1598988579883,"z":3},{"ev":"T","sym":"AAOI","i":"1452","x":12,"p":11.63,"s":9,"c":[14,37,41],"t":1598988579917,"z":3}]
I, [2020-09-01T15:29:39.945616 #37834]  INFO -- : [{"ev":"T","sym":"BKI","i":"52983559697462","x":10,"p":85.64,"s":100,"c":[14,41],"t":1598988579855,"z":1},{"ev":"T","sym":"BKI","i":"52983559697462","x":10,"p":85.64,"s":100,"c":[14,41],"t":1598988579855,"z":1},{"ev":"T","sym":"NWE","i":"62879871164727","x":12,"p":51.545,"s":17,"c":[37],"t":1598988579883,"z":1},{"ev":"T","sym":"NWE","i":"62879871164727","x":12,"p":51.545,"s":17,"c":[37],"t":1598988579883,"z":1}]
I, [2020-09-01T15:29:39.954029 #37834]  INFO -- : [{"ev":"T","sym":"FORM","i":"361","x":2,"p":26.74,"s":23,"c":[14,37,41],"t":1598988579900,"z":3},{"ev":"T","sym":"FORM","i":"153","x":10,"p":26.74,"s":77,"c":[14,37,41],"t":1598988579900,"z":3}]
I, [2020-09-01T15:29:39.954268 #37834]  INFO -- : [{"ev":"T","sym":"KNX","i":"54272018510146","x":15,"p":46.36,"s":100,"t":1598988579932,"z":1}]
I, [2020-09-01T15:29:39.954636 #37834]  INFO -- : [{"ev":"T","sym":"CLI","i":"71688443282231","x":4,"p":12.365,"s":100,"t":1598988579925,"z":1},{"ev":"T","sym":"CLI","i":"71688443282231","x":4,"p":12.365,"s":100,"t":1598988579925,"z":1}]
I, [2020-09-01T15:29:39.954758 #37834]  INFO -- : [{"ev":"T","sym":"GES","i":"71688443282233","x":4,"p":11.9569,"s":17,"c":[10,37,41],"t":1598988579941,"z":1},{"ev":"T","sym":"GES","i":"71688443282233","x":4,"p":11.9569,"s":17,"c":[10,37,41],"t":1598988579941,"z":1}]
I, [2020-09-01T15:29:39.955124 #37834]  INFO -- : [{"ev":"T","sym":"RYN","i":"79372493076529","x":4,"p":29.2136,"s":10,"c":[37],"t":1598988579939,"z":1},{"ev":"T","sym":"RYN","i":"79372493076529","x":4,"p":29.2136,"s":10,"c":[37],"t":1598988579939,"z":1}]
I, [2020-09-01T15:29:39.955658 #37834]  INFO -- : [{"ev":"T","sym":"SDC","i":"3765","x":4,"p":7.775,"s":1,"c":[37],"t":1598988579904,"z":3}]

[jrbell19]

We have implemented much more logic to filter this duplicates out of the streams, so you should no longer see these duplicates. It is still possible, but the likelihood is extremely low. Over the past week we have only seen ~15 duplicates out of the billions of messages broadcasted.

If you begin to see more and more duplicates, please let me know and we will investigate.