Cancel CEX orders when CEX price feed is down, start up as normal once price feed is back

[kilrau]

CEX price stream lost for more than 10 seconds -> cancel CEX & OpenDEX orders, infinitely retry cancelling if one happens to fail, retry connecting to the CEX price stream after successful cancellation, re-issue orders as on startup once price stream is back

Arby should NOT exit.

[ghost]

The above makes sense except:

Placing CEX order fails for more than 10 seconds (e.g. a rate limit) -> cancel OpenDEX orders and retry placing CEX order

CCXT should prevent us from ever getting rate limited. Think we can keep retrying until X times and afterwards exit with code 1 (trigger the non-0 protocol).

[kilrau]

Agree, adjusted the above.

Can we tackle the first point first? I added retrying connection to price stream. Loss of price stream seems to be a common issue and as of now arby doesn't seem to retry and simply exits with 0 after 12 hours:

2020-07-06 07:47:38.3838 [Centralized] info: Base asset balance 123 and quote asset balance 321
2020-07-06 07:47:39.3939 [OpenDEX] trace: Base asset balance 13.395 (maxinbound: 90071992.54740991, maxoutbound: 13.395) and quote asset balance 2.4308062 (maxinbound: 2.51910329, maxoutbound: 2.3808062).
Error: Price feed lost
2020-07-06 07:47:52.5252 [Centralized] trace: ETHBTC stream closed
2020-07-06 19:47:53,773 INFO exited: arby (exit status 0; expected)

[kilrau]

As just discussed with @erkarl , we separated the original issue list into 3 separate issues:

• this one. Only thing missing: cancel CEX orders too & retry cancelling -> needed for release
• https://github.com/ExchangeUnion/market-maker-tools/issues/46 -> needed for release
• https://github.com/ExchangeUnion/market-maker-tools/issues/45 -> next project since edge case