Crashing after every filled order (live)

[Fransenson]

Crashing after order is filled with the following exception:

line 252, in retry
    raise Exception("Max retries on %s (%s) hit, raising." % (path, json.dumps(postdict or '')))
Exception: Max retries on order/bulk ({"orders": [{"side": "Buy", "clOrdID": "mm_bitmex_c59XCcvmQj6tGHl8jkABLA", "symbol": "XBTUSD", "price": 7360.5, "orderQt
y": 100}]}) hit, raising.

Can I raise the max-retries anywhere to prevent this?

[ch3rryzzz]

Same problem, readed all posts, increased timeout - nothing helps. Dont know how to fix it :(

[ryanfox]

Are you having network issues with other sites? Are any requests to the BitMEX API successful? Does the web interface load and work normally?

Increasing the timeout is usually the first place to go. If that still doesn't solve the issue, likely you have larger problems.

[verata-veritatis]

EDIT 13-09-18:

The comments I highlighted at the bottom specifically state that POST/PUT should not be retried; however, amend_bulk_orders seems to do this pretty well, even after a 503. I have modified the create_bulk_orders function in bitmex.py (line 171) to return rethrow_errors = True. Will update on progress.

---

Hi Ryan, since no one else is participating here, I thought I would add some information.

The script has absolutely no problem cancelling and submitting orders in bulk upon initiation, reset, or exit, but as soon as a limit order is filled, the bot attempts to create a new order and crashes every time with 503 without a single retry attempt:

raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://www.bitmex.com/api/v1/order/bulk). 

...

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/bitmex.py", line 252, in retry
    raise Exception("Max retries on %s (%s) hit, raising." % (path, json.dumps(postdict or '')))
Exception: Max retries on order/bulk ({"orders": [{"price": [HIDDEN], "orderQty": [HIDDEN], "side": "Sell", "clOrdID": "mm_bitmex_[HIDDEN]", "symbol": "XBTUSD"}]}) hit, raising.

Even after changing the number of max_retries in bitmex.py (line 238), and increasing the TIMEOUT setting, I still get the following:

[HIDDEN] - WARNING - bitmex - Unable to contact the BitMEX API (503), retrying. Request: https://www.bitmex.com/api/v1/order/bulk 
 {"orders": [{"price": [HIDDEN], "orderQty": [HIDDEN], "side": "Sell", "clOrdID": "mm_bitmex_[HIDDEN]", "symbol": "XBTUSD"}]}
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/bitmex.py", line 263, in _curl_bitmex
    response.raise_for_status()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: https://www.bitmex.com/api/v1/order/bulk

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/marketmaker", line 11, in <module>
    sys.exit(run())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/__init__.py", line 22, in run
    market_maker.run()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/market_maker.py", line 624, in run
    om.run_loop()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/market_maker.py", line 593, in run_loop
    self.place_orders()  # Creates desired orders and converges to existing orders
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/market_maker.py", line 402, in place_orders
    return self.converge_orders(buy_orders, sell_orders)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/market_maker.py", line 488, in converge_orders
    self.exchange.create_bulk_orders(to_create)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/market_maker.py", line 193, in create_bulk_orders
    return self.bitmex.create_bulk_orders(orders)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/bitmex.py", line 104, in wrapped
    return fn(self, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/bitmex.py", line 178, in create_bulk_orders
    return self._curl_bitmex(path='order/bulk', postdict={'orders': orders}, verb='POST')
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/bitmex.py", line 313, in _curl_bitmex
    return retry()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/market_maker/bitmex.py", line 252, in retry
    raise Exception("Max retries on %s (%s) hit, raising." % (path, json.dumps(postdict or '')))
Exception: Max retries on order/bulk ({"orders": [{"price": [HIDDEN], "orderQty": [HIDDEN], "side": "Sell", "clOrdID": "mm_bitmex_[HIDDEN]", "symbol": "XBTUSD"}]}) hit, raising.

From the comments in the python script, it seems it has been touched on (line 233, bitmex.py):

        # By default don't retry POST or PUT. Retrying GET/DELETE is okay because they are idempotent.
        # In the future we could allow retrying PUT, so long as 'leavesQty' is not used (not idempotent),
        # or you could change the clOrdID (set {"clOrdID": "new", "origClOrdID": "old"}) so that an amend
        # can't erroneously be applied twice.

Obviously, order creation is a POST request. Reading this makes me assume that by default, POST requests do not retry, and thus cause a crash at the first instance, but I can't seem to force the script to retry...

I know of three others using the Sample Market Maker and they're all running into the same issues.

[ajmanser]

@anakratis did this fix the problem? Im trying myself now too

[verata-veritatis]

@anakratis did this fix the problem? Im trying myself now too

This did not work but a simple exception condition did. I'm using a custom script so this might be different for you (since self.prepare_orders() will spit out different orders each time it's run). You could wrap the entire converge_orders function with an exception.

I wrapped my self.converge_orders() like so:

while True:
     try:
          self.converge_orders(buy_order, sell_order)
     except:
          logger.info("HTTPError raised. Retrying in 2 seconds...")
          sleep(2)
          continue
     break

In this case, I have my own bulk order lists set as buy_order and sell_order. I send these to the converge_orders function, and wrap it with an error handler that will keep trying to submit to BitMEX endlessly. This is the result:

2018-09-26 [HIDDEN] - INFO - market_maker - Creating 1 orders:
2018-09-26 [HIDDEN] - INFO - market_maker -  Buy [HIDDEN] @ 6529.5
2018-09-26 [HIDDEN] - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order/bulk: {"orders": [{"price": 6529.5, "orderQty": [HIDDEN], "side": "Buy", "clOrdID": "[HIDDEN]", "symbol": "XBTUSD"}]}
2018-09-26 [HIDDEN] - WARNING - bitmex - Unable to contact the BitMEX API (503), retrying. Request: https://www.bitmex.com/api/v1/order/bulk 
 {"orders": [{"price": 6529.5, "orderQty": [HIDDEN], "side": "Buy", "clOrdID": "[HIDDEN]", "symbol": "XBTUSD"}]}
2018-09-26 [HIDDEN] - INFO - market_maker - HTTPError raised. Retrying in 2 seconds...
2018-09-26 [HIDDEN] - INFO - market_maker - Creating 1 orders:
2018-09-26 [HIDDEN] - INFO - market_maker -  Buy [HIDDEN] @ 6529.5
2018-09-26 [HIDDEN] - INFO - bitmex - sending req to https://www.bitmex.com/api/v1/order/bulk: {"orders": [{"price": 6529.5, "orderQty": [HIDDEN], "side": "Buy", "clOrdID": "[HIDDEN]", "symbol": "XBTUSD"}]}
2018-09-26 [HIDDEN] - INFO - ws_thread - Execution: Buy [HIDDEN] Contracts of XBTUSD at 6529.5

[ajmanser]

@anakratis thank you so much! Will give this a shot today

[darythofc]

thanks for the workaround @anakratis , this worked well for me.

[rascao]

hi @anakratis any noob guide how to Workaround this? i really don't understand how you make it to work

[ghost]

Same problem here, its not retrying. Any ideas for a quick fix ?