1 minute bars are missing or duplicated

[openSourceBugs]

watch(symbol=ticker, interval_len=60, interval_type='s', update=1, lookback_bars=600)

The above method is random and duplicates bars at :00 and :59 of each minute, or sometimes completely skips bars due to random timing. As such, this method is completely useless for any sort of real time applications.

[openSourceBugs]

It is definitely broken. This output is taken from the exmaple.py run with -i as an option (interval_data) 18.0 42.0 0.0 Bar Listener: Process live bar update: [(b'SPY', '2023-10-19', 67320000000, 425.8, 425.8, 425.79, 425.79, 121207055, 698, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.74, 425.74, 121207577, 496, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.72, 425.72, 121207941, 774, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.72, 425.72, 121208241, 1074, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.72, 425.73, 121208341, 1174, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.8, 425.72, 425.8, 121210498, 3274, 0)] 18.0 44.0 0.0 Bar Listener: Process live bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.8, 425.72, 425.8, 121210498, 3274, 0)] 18.0 46.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67560000000, 425.69, 425.69, 425.69, 425.69, 121211639, 1000, 0)]

We can clearly see that it's skipping some minutes.

[akapur]

pyiqfeed sends you what IQFeed sends it. It doesn't duplicate bars or drop them. This is likely an iqfeed problem or maybe a flaky network on your end. Note that IQFeed will not send you a bar. I don't think that was necessarily the case here, given that it was SPY but it's probably something you should check first.

Also note that with any sort of networked callback based API like this it's generally a good idea to design your application on the assumption that you can get duplicate messages and of course if there are hiccups, messages can be skipped too. Exactly once delivery of messages is very hard and most market data apis won't even try to get there. Most market data api's aim for at least once but even that is hard and can cause latency and in most cases you'd rather have a skipped message or two than crazy latency so it's "sorta kinda" at least once.

If you get data directly from an exchange, you usually get messages via udp, so not even tcp guarantees. You do get a sequence number so if you figure out you missed something, you can ask for a retransmit. You can sort of approximate this with iqfeed by requesting "historical" data for the bars you missed even if it was a minute or two ago.

Good luck.

On Thu, Oct 19, 2023 at 6:47 PM openSourceBugs @.***> wrote:

It is definitely broken. This output is taken from the exmaple.py run with -i as an option (interval_data) 18.0 42.0 0.0 Bar Listener: Process live bar update: [(b'SPY', '2023-10-19', 67320000000, 425.8, 425.8, 425.79, 425.79, 121207055, 698, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.74, 425.74, 121207577, 496, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.72, 425.72, 121207941, 774, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.72, 425.72, 121208241, 1074, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.74, 425.72, 425.73, 121208341, 1174, 0)] 18.0 44.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.8, 425.72, 425.8, 121210498, 3274, 0)] 18.0 44.0 0.0 Bar Listener: Process live bar update: [(b'SPY', '2023-10-19', 67440000000, 425.74, 425.8, 425.72, 425.8, 121210498, 3274, 0)] 18.0 46.0 0.0 Bar Listener: Process latest bar update: [(b'SPY', '2023-10-19', 67560000000, 425.69, 425.69, 425.69, 425.69, 121211639, 1000, 0)]

We can clearly see that it's skipping some minutes.

— Reply to this email directly, view it on GitHub https://github.com/akapur/pyiqfeed/issues/65#issuecomment-1771806150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGJAVAFYIFJRAEPPXH4KPDYAGUWRAVCNFSM6AAAAAA6HY3MNCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRHAYDMMJVGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[openSourceBugs]

Is there a way to do this without polling or the timing issues? Making multiple requests due to a missed timing seems terrible. I can certainly retry the request_bars function over and over again until the missing time's data is correct, but this seems wasteful and has bad performance.

[openSourceBugs]

For now, i'm going to poll the request_bars function with multiple threads/processes, check the data for differences and way down the road colocate/switch to another data provider if I ever get there.