Timing out and not reconnecting

[Mexflubber]

Similar to https://github.com/happydasch/btoandav20/issues/54

I am experiencing timeouts but it my case it never recovers. Waited for 10 minutes and this was the last of it. Dis happened for at least 3-4 hours. I tried debugging v20 without success.

Data0, 0492, 737672.471817, 2020-09-04T11:19:25.000000, 1.184910, 1.184910, 1.184830, 1.184830,     11, 0, 1.184932
Data0, 0493, 737672.471875, 2020-09-04T11:19:30.000000, 1.184840, 1.184900, 1.184800, 1.184800,     11, 0, 1.184894
Data0, 0494, 737672.471933, 2020-09-04T11:19:35.000000, 1.184800, 1.184800, 1.184780, 1.184790,      8, 0, 1.184860
Data0, 0495, 737672.471991, 2020-09-04T11:19:40.000000, 1.184780, 1.184800, 1.184770, 1.184800,     11, 0, 1.184828
Data0, 0496, 737672.472049, 2020-09-04T11:19:45.000000, 1.184810, 1.184900, 1.184810, 1.184900,     20, 0, 1.184824
Data0, 0497, 737672.472106, 2020-09-04T11:19:50.000000, 1.184900, 1.184930, 1.184880, 1.184920,     10, 0, 1.184842
Data0, 0498, 737672.472164, 2020-09-04T11:19:55.000000, 1.184930, 1.184980, 1.184930, 1.184960,     11, 0, 1.184874
Data0, 0499, 737672.472222, 2020-09-04T11:20:00.000000, 1.184940, 1.185000, 1.184940, 1.184980,      9, 0, 1.184912
Data0, 0500, 737672.472280, 2020-09-04T11:20:05.000000, 1.185000, 1.185060, 1.185000, 1.185060,     12, 0, 1.184964
***** DATA NOTIF: LIVE
Data0, 0501, 737672.472331, 2020-09-04T11:20:09.433757, 1.185055, 1.185055, 1.185055, 1.185055,      0, 0, 1.184995
Data0, 0502, 737672.472338, 2020-09-04T11:20:09.997282, 1.185065, 1.185065, 1.185065, 1.185065,      0, 0, 1.185024
Data0, 0503, 737672.472342, 2020-09-04T11:20:10.391929, 1.185055, 1.185055, 1.185055, 1.185055,      0, 0, 1.185043
Data0, 0504, 737672.472356, 2020-09-04T11:20:11.576372, 1.185070, 1.185070, 1.185070, 1.185070,      0, 0, 1.185061
Data0, 0505, 737672.472358, 2020-09-04T11:20:11.767691, 1.185110, 1.185110, 1.185110, 1.185110,      0, 0, 1.185071
Data0, 0506, 737672.472370, 2020-09-04T11:20:12.736052, 1.185095, 1.185095, 1.185095, 1.185095,      0, 0, 1.185079
Data0, 0507, 737672.472372, 2020-09-04T11:20:12.933878, 1.185095, 1.185095, 1.185095, 1.185095,      0, 0, 1.185085
Data0, 0508, 737672.472381, 2020-09-04T11:20:13.719048, 1.185115, 1.185115, 1.185115, 1.185115,      0, 0, 1.185097
***** DATA NOTIF: CONNBROKEN
***** DATA NOTIF: DELAYED
***** STORE NOTIF: v20 REST request to https://stream-fxpractice.oanda.com:443/v3/accounts/101-001-XXX426-001/pricing/stream?instruments=EUR_USD has timed out (stream)

[Mexflubber]

After doing some debugging I found that this always happens after a "Delayed", and the line that breaks is

msg = self.qhist.get() of https://github.com/happydasch/btoandav20/blob/master/btoandav20/feeds/oandav20feed.py#L436

checked the size of the queue and it is 0.

[happydasch]

DATA NOTIF: CONNBROKEN DATA NOTIF: DELAYED ***** STORE NOTIF: v20 REST request to https://stream-fxpractice.oanda.com:443/v3/accounts/x/pricing/stream?instruments=EUR_USD has timed out (stream)

this just tells you, that there was a disconnect. this may happen because of a timeout, you internet connection goes away, ...

for the queue, this just means, it is still waiting for a reconnection. The code will continue, after the reconnection was successful. Until then, it will wait at this point.

What else would you expect to happen at the point, where no connection to the stream is available?

Also check out the param quicknotify=True when running backtrader. This will allow to show notifications, while there is no new data.

Don't get me wrong, if there is a issue, i would fix it, but this behaviour is inteded.

[happydasch]

It could be, that there is another underlaying problem in the store, where the reconnection is happening. But i cannot reproduce this behaviour.

[happydasch]

also, with the practice account, it is from time to time not possible to connect, the timeouts occour way more often then in live trading.

[Mexflubber]

Well, the code is dying when it gets to this line and there is a TODO here which I guess has something to do with it.

The what I am seeing is that there is no connection retry. Not even after 5 hours. But if I run the code in a different terminal it starts to work again.

[happydasch]

The lines you referenced are fine. The TODO i added for the param onlyComplete=False. This just means, when fetching data from oanda, which happened between reconnections will include incomplete candles. So you would even get a new candle within the given compression / timeframe.

That the code waits at the empty queue is also fine, since it is waiting for a reconnection.

If a reconnection does not happen, then it will be at a different place (specially in the store, where historical candles are fetched).

If you are fine debugging, check this block: https://github.com/happydasch/btoandav20/blob/4624873b87df05d27db62a2ae5faae8377449e01/btoandav20/stores/oandav20store.py#L718 especially https://github.com/happydasch/btoandav20/blob/4624873b87df05d27db62a2ae5faae8377449e01/btoandav20/stores/oandav20store.py#L745 and https://github.com/happydasch/btoandav20/blob/4624873b87df05d27db62a2ae5faae8377449e01/btoandav20/stores/oandav20store.py#L731

[Mexflubber]

So, I kept debugging and on this line it appears that an exception is being swallowed here.

After debugging I get the following error. I might need more help from you. If you want we can keep chatting over email at aldogarcia at gmail and we can post the resolution here after (I can even send a merge request).

`GET https://api-fxpractice.oanda.com:443/v3/instruments/EUR_USD/candles?price=M&granularity=S5&from=2020-09-05T03%3A32%3A00.000000000Z&includeFirst=True expected status 200, got 400 (Bad Request)"

[happydasch]

i've send you an email

[happydasch]

For anyone having issues with reconnection, in this case, it worked to provide a timezone. I will look into that more deeply. For now, if reconnections are not working, try to set a timezone.

Ex. python examples/oandav20test/oandav20test.py --token XXX --account YYY --data0 EUR_USD --timezone America/Los_Angeles