robust websocket connectivity

[alanfuji]

I've had many issues maintaining consistent websocket connections to okcoin.cn. I'm aware that it's just not easy, with the great wall of China and whatnot. The problem I'm about to describe is one that I've also had with a completely different websocket application I was working on in C++.

Sometimes, no errors are thrown and there's no sign that the connection has dropped, except that there's no data coming in. It might start coming back after a few seconds, a few minutes, or maybe never. This can lead to many identical candles if the public websocket is down long enough, and it wrongly thought its trade succeeded when the private websocket went down (I think that's why anyway.)

I've been playing around with two different approaches to monitoring the two connections. The usual way is to send {'event':'ping'} on the websocket occasionally. For the public connection, you can also just keep track of the time since the last ticker data and reset the connection if it exceeds a threshold. It seems to be working so far. I'm testing it by disabling and enabling my wifi while it's running, and it reconnects when it can - it may be a good idea to pause trading and re-initialize the indicators if it went on for long enough, but it's not so hard to notice the problem and reconnect fast enough that it usually happens between candles. I'll try running it overnight with 1-minute candles.

By the way, is there a good reason to use two different websocket(s) libraries?

cheers alan

[ghost]

I would like to expand on this thread. In my simulator I have noticed this a couple times and usually just close it and reopen. The last candle that was recorded before websocket connectivity gets repeated until infinity basically until you close and restart the application:

Candle: 2921 | Price: 1511.34 cny | Time: 23:36:57 | Date: 2015/03/31 MACD : We are in a Downtrend | Diff: -0.20117133906319395 MACD Hist: -0.36800173954486864 DMI: Not yet enough data to calculate Candle: 2922 | Price: 1497.35 cny | Time: 23:51:57 | Date: 2015/03/31 MACD : We are in a Downtrend | Diff: -4.9990519531738755 MACD Hist: -4.558131488519603 DMI: Not yet enough data to calculate Candle: 2923 | Price: 1503.48 cny | Time: 00:06:57 | Date: 2015/04/01 MACD : We are in a Downtrend | Diff: -4.538247879622077 MACD Hist: -3.6152888955598277 DMI: Not yet enough data to calculate Candle: 2924 | Price: 1503.48 cny | Time: 00:21:57 | Date: 2015/04/01 MACD : We are in a Downtrend | Diff: -3.9680588590069874 MACD Hist: -2.6868528308335926 DMI: Not yet enough data to calculate Candle: 2925 | Price: 1503.48 cny | Time: 00:36:57 | Date: 2015/04/01 MACD : We are in a Downtrend | Diff: -3.399808528833546 MACD Hist: -1.86935514764131 DMI: Not yet enough data to calculate Candle: 2926 | Price: 1503.48 cny | Time: 00:51:57 | Date: 2015/04/01 MACD : We are in a Downtrend | Diff: -2.879720180525055 MACD Hist: -1.1905295288230757 DMI: Not yet enough data to calculate Candle: 2927 | Price: 1503.48 cny | Time: 01:06:57 | Date: 2015/04/01 MACD : We are in a Downtrend | Diff: -2.4230400203528006 MACD Hist: -0.6475141488095482 DMI: Not yet enough data to calculate Candle: 2928 | Price: 1503.48 cny | Time: 01:21:57 | Date: 2015/04/01 MACD : We are in a Downtrend | Diff: -2.030836618584999 MACD Hist: -0.22527418856624704 DMI: Not yet enough data to calculate Candle: 2929 | Price: 1503.48 cny | Time: 01:36:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -1.6981821496328848 MACD Hist: 0.09474730622282412 DMI: Not yet enough data to calculate SIMULATOR: BUYING 0.973 btc at 1503.48 cny SIMULATOR: Asset: 0.9829999999999998 btc, Currency: 15.159340000000157 cny, Net Worth: 1493.08018 cny Candle: 2930 | Price: 1503.48 cny | Time: 01:51:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -1.418060941766953 MACD Hist: 0.33076633596066696 DMI: Not yet enough data to calculate Candle: 2931 | Price: 1503.48 cny | Time: 02:06:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -1.1831720266754928 MACD Hist: 0.4991075744577591 DMI: Not yet enough data to calculate Candle: 2932 | Price: 1503.48 cny | Time: 02:21:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.9867039764976653 MACD Hist: 0.6137431982078705 DMI: Not yet enough data to calculate Candle: 2933 | Price: 1503.48 cny | Time: 02:36:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.8226169575489166 MACD Hist: 0.6863207798440758 DMI: Not yet enough data to calculate Candle: 2934 | Price: 1503.48 cny | Time: 02:51:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.6856959531910434 MACD Hist: 0.726389809589955 DMI: Not yet enough data to calculate Candle: 2935 | Price: 1503.48 cny | Time: 03:06:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.5715042052761419 MACD Hist: 0.7416896095631087 DMI: Not yet enough data to calculate Candle: 2936 | Price: 1503.48 cny | Time: 03:21:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.47629895987211057 MACD Hist: 0.7384366367357118 DMI: Not yet enough data to calculate Candle: 2937 | Price: 1503.48 cny | Time: 03:36:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.39693852763093673 MACD Hist: 0.7215856490972521 DMI: Not yet enough data to calculate Candle: 2938 | Price: 1503.48 cny | Time: 03:51:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.33079347022794536 MACD Hist: 0.6950565057355089 DMI: Not yet enough data to calculate Candle: 2939 | Price: 1503.48 cny | Time: 04:06:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.2756669071241049 MACD Hist: 0.6619262372111906 DMI: Not yet enough data to calculate Candle: 2940 | Price: 1503.48 cny | Time: 04:21:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.22972526357057177 MACD Hist: 0.6245893065571092 DMI: Not yet enough data to calculate Candle: 2941 | Price: 1503.48 cny | Time: 04:36:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.19143914012556706 MACD Hist: 0.5848900852959829 DMI: Not yet enough data to calculate Candle: 2942 | Price: 1503.48 cny | Time: 04:51:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.15953332701315048 MACD Hist: 0.5442316750662348 DMI: Not yet enough data to calculate Candle: 2943 | Price: 1503.48 cny | Time: 05:06:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.1329447942985098 MACD Hist: 0.5036648892184196 DMI: Not yet enough data to calculate Candle: 2944 | Price: 1503.48 cny | Time: 05:21:57 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.11078750614251476 MACD Hist: 0.46396074474213056 DMI: Not yet enough data to calculate Candle: 2945 | Price: 1503.48 cny | Time: 05:36:58 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.09232301056545111 MACD Hist: 0.42566932969340665 DMI: Not yet enough data to calculate Candle: 2946 | Price: 1503.48 cny | Time: 05:51:58 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.0769358865279628 MACD Hist: 0.3891674591743191 DMI: Not yet enough data to calculate Candle: 2947 | Price: 1503.48 cny | Time: 06:06:58 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.06411326096849734 MACD Hist: 0.3546971335886334 DMI: Not yet enough data to calculate Candle: 2948 | Price: 1503.48 cny | Time: 06:21:58 | Date: 2015/04/01 MACD : We are in an Uptrend | Diff: -0.053427728571250555 MACD Hist: 0.32239646998754135 DMI: Not yet enough data to calculate

[Sim-fo]

Incidentally, about the DMI. I tried it in full mode. The indicator doesn't work: Candle: 3441 ... DMI: Not yet enough data to calculate

[RealJohnGalt]

Apologies for the late responses. @Sim-fo I'll fix DMI's Full mode shortly. @alanfuji: One websocket library is asyncio only. It gets the ticker data in an always running asyncio event loop. For private operations (if live trading is enabled), it uses a standard websocket library.

I just did a small change I'm testing which pings every 15 (configurable) ticker runs. If it doesn't get a response or runs into an exception, it attempts a reconnect, and sleeps the coroutine to block new runs (which would make it attempt another connection over and over again if using very small candle sizes).

[RealJohnGalt]

fb2e2758b88 is that change. It still needs more testing, but it looks good so far (despite being ugly code).

[RealJohnGalt]

There are newer commits in the staging branch to additionally workaround this issue. It's been cleaned up quite a bit, and now semi-gracefully recovers. Here's an example with 3s candles, API set to verbose, and the reconnect wait set to 12s (so four 3s candles are using old data in this case):

Candle: 153 | Price: 1460.43 cny | Time: 05:15:43 | Date: 2015/04/12
Candle: 154 | Price: 1460.46 cny | Time: 05:15:46 | Date: 2015/04/12
Reconnecting to Public OKCoin WebSocket.
Candle: 155 | Price: 1460.38 cny | Time: 05:15:49 | Date: 2015/04/12
Candle: 156 | Price: 1460.38 cny | Time: 05:15:52 | Date: 2015/04/12
Candle: 157 | Price: 1460.38 cny | Time: 05:15:55 | Date: 2015/04/12
Candle: 158 | Price: 1460.38 cny | Time: 05:15:58 | Date: 2015/04/12
Candle: 159 | Price: 1460.46 cny | Time: 05:16:01 | Date: 2015/04/12
Candle: 160 | Price: 1460.49 cny | Time: 05:16:04 | Date: 2015/04/12