Failed Test #356 - Unable to cancel orders on client interface

[mrhzysbl]

Summary and Screenshots

Flooded with "failed to cancel orders" after getting a connection timed out message. The orders were no longer on the web application and even after 3 hours, the client is still trying to cancel orders.

The orders were created at 2023-05-24 07:21:58,735

2023-05-24 08:02:46,717 - 1381 - hummingbot.connector.exchange.polkadex.polkadex_exchange.PolkadexExchange - ERROR - Failed to cancel order HBOTBPXCT5fc64a7481468423dae19d0
Traceback (most recent call last):
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_socket.py", line 108, in recv
    bytes_ = _recv()
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_socket.py", line 87, in _recv
    return sock.recv(bufsize)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/ssl.py", line 1226, in recv
    return self.read(buflen)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/ssl.py", line 1101, in read
    return self._sslobj.read(len)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/admin/polka2/hummingbot/connector/exchange_py_base.py", line 549, in _execute_order_cancel
    cancelled = await self._execute_order_cancel_and_process_update(order=order)
  File "/Users/admin/polka2/hummingbot/connector/exchange_py_base.py", line 569, in _execute_order_cancel_and_process_update
    cancelled = await self._place_cancel(order.client_order_id, order)
  File "/Users/admin/polka2/hummingbot/connector/exchange/polkadex/polkadex_exchange.py", line 147, in _place_cancel
    await self._data_source.cancel_order(
  File "/Users/admin/polka2/hummingbot/connector/exchange/polkadex/polkadex_data_source.py", line 330, in cancel_order
    cancel_request = self._substrate_interface.create_scale_object("H256").encode(order.exchange_order_id)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/substrateinterface/base.py", line 1348, in create_scale_object
    self.init_runtime(block_hash=block_hash)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/substrateinterface/base.py", line 648, in init_runtime
    block_hash = self.get_chain_head()
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/substrateinterface/base.py", line 435, in get_chain_head
    response = self.rpc_request("chain_getHead", [])
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/substrateinterface/base.py", line 323, in rpc_request
    self.__rpc_message_queue.append(json.loads(self.websocket.recv()))
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_core.py", line 362, in recv
    opcode, data = self.recv_data()
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_core.py", line 385, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_core.py", line 406, in recv_data_frame
    frame = self.recv_frame()
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_core.py", line 445, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_abnf.py", line 338, in recv_frame
    self.recv_header()
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_abnf.py", line 294, in recv_header
    header = self.recv_strict(2)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_abnf.py", line 373, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_core.py", line 529, in _recv
    return recv(self.sock, bufsize)
  File "/opt/anaconda3/envs/hummingbot/lib/python3.8/site-packages/websocket/_socket.py", line 110, in recv
    raise WebSocketTimeoutException("Connection timed out")
websocket._exceptions.WebSocketTimeoutException: Connection timed out

Steps To Reproduce

1. Install and clone featured branch
2. Create PMM strategy with fast refresh rate
3. Start strategy and monitor logs

Expected Results

The client should be able to recover from failed order cancellation since the orders were no longer found on the web app.

Test Result Comparison

n/a

Attachments

Please attach your config file and log file for debugging. Polkadex.zip

[mrhzysbl]

No longer seeing failed to cancel orders so far. Closing ticket.