Open dfelton opened 2 years ago
Note: the below verbiage is written under the context that we are only trading on "USD" quoted pairs. However, problem is quote currency agnostic.
After today's 9 hour service outage Gemini encountered which resulted in the cancellation of all limit orders on the exchange it brought to light the importance of handling this properly for partially filled orders. Up until now I have been resolving this scenario by performing the following manual steps:
TradeRepeater
suite of classes are not running (touch var/emergency_shutdown
if necessary to halt trading)BUY_PLACED
status, reset the record by manipulating it in the following way:
SET status = "BUY_READY", buy_order_id = null, buy_client_order_id = NULL, meta = null
SELL_PLACED
status, reset the record by manipulating it in the following way:
SET status = "BUY_FILLED", sell_order_id = null, sell_client_order_id = NULL, meta = '{"buy_price":"1234.56"}'
meta
column, the sell_price
would be unset, and the buy_price
value of the original buy for the record left alone)This worked well enough for the time being. It was already known that there was the possibility that some orders had been partially filled. For these situations, the following was accepted:
usd
bucket, else the reserve funds that are used for the temporary hold of transaction fees beyond the current maker API fees that get returned to available balance upon order fill (no automated tracking here, manual execution of the bin/gemini repeater:account
command would yield the necessary information to determine if any manual re-allocation of bucket funds would be necessary afterwards).amount
and amount available
records from the exchange)What had not been considered was the following:
MarketClosed
event on the exchange's entire set of order books across all pairs.
In today's MarketClosed
of all Limit Orders on the exchange, the bot put itself into a situation where it was $40 away from running out of USD funds for more buy orders. Furthermore, because all shibusd
records had been configured to never hodl any of the profits of coin's sales into profit buckets, the last record it attempted to re-place a sell order for, encountered an insufficient funds error (Attempted to place a sell order of 100,000
SHIB, however only 65983.79777000
was available. Indicating that roughly 34,000 SHIB had already been sold in prior partially filled sell orders).
At times when gemini performs maintenance on their servers they will close all open limit orders an a trading pair. When this happens, the application does not recognize that the order is not live anymore and will not re-place the original buy / sell order.
Example order status after this event has occurred: