Failed Test PR #315 - Failed to cancel orders using Injective

[rxlxrxsx]

Summary and Screenshots

There are instances that orders are failing to cancel and may take a couple of cycles before being successfully canceled.

2023-03-13 11:15:50,318 - 12105 - hummingbot.connector.gateway.clob_spot.gateway_clob_spot.GatewayCLOBSPOT - ERROR - Failed to cancel order BIJUT5f6cb3fd0f02ed6e3f2509794b922eaf289e249424ddd
Traceback (most recent call last):
  File "/Users/reiniersalazar/hbot-ca-dexalot/hummingbot/connector/exchange_py_base.py", line 552, in _execute_order_cancel
    cancelled = await self._execute_order_cancel_and_process_update(order=order)
  File "/Users/reiniersalazar/hbot-ca-dexalot/hummingbot/connector/gateway/clob_spot/gateway_clob_spot.py", line 668, in _execute_order_cancel_and_process_update
    cancelled, misc_order_updates = await self._api_data_source.cancel_order(order=order)
  File "/Users/reiniersalazar/hbot-ca-dexalot/hummingbot/connector/gateway/clob_spot/data_sources/injective/injective_api_data_source.py", line 281, in cancel_order
    cancelation_result = await self._get_gateway_instance().clob_cancel_order(
  File "/Users/reiniersalazar/hbot-ca-dexalot/hummingbot/core/gateway/gateway_http_client.py", line 1146, in clob_cancel_order
    resp = await self.api_request(method="delete", path_url="clob/orders", params=request_payload)
  File "/Users/reiniersalazar/hbot-ca-dexalot/hummingbot/core/gateway/gateway_http_client.py", line 218, in api_request
    raise e
  File "/Users/reiniersalazar/hbot-ca-dexalot/hummingbot/core/gateway/gateway_http_client.py", line 206, in api_request
    raise ValueError(f"Error on {method.upper()} {url} Error: {parsed_response}")
ValueError: Error on DELETE https://localhost:15888/clob/orders Error: {'message': 'Unknown error.', 'httpErrorCode': 503, 'errorCode': 1099, 'stack': 'Error: Request failed with status code 503\n    at createError (/Users/reiniersalazar/gateway-dexalot/node_modules/axios/lib/core/createError.js:16:15)\n    at settle (/Users/reiniersalazar/gateway-dexalot/node_modules/axios/lib/core/settle.js:17:12)\n    at IncomingMessage.handleStreamEnd (/Users/reiniersalazar/gateway-dexalot/node_modules/axios/lib/adapters/http.js:269:11)\n    at IncomingMessage.emit (node:events:539:35)\n    at endReadableNT (node:internal/streams/readable:1344:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)'}
2023-03-13 11:15:50,605 - 12105 - hummingbot.core.gateway.gateway_http_client - WARNING - Call to https://localhost:15888/clob/orders failed. See logs for more details.

Steps To Reproduce

1. Clone and install branch
2. Setup and run gateway using the feature PR for Dexalot
3. Connect Injective wallet
4. Create and run a pure MM strategy using Injective
5. Wait until the bot tries to cancel the order

Test Result Comparison

I tried replicating this on the foundation repository however I encountered an issue which further blocks testing.

Attachments

Logs and config: inective_315.zip

[petioptrv]

I am not able to replicate this issue after the latest commits. @rxlxrxsx, can you still replicate it on your end?

[rxlxrxsx]

I'm still getting this issue using the latest commit 0f46fb7. So far I was only able to replicate it once after restarting the strategy.

Logs and config: injective.zip

[rxlxrxsx]

As discussed with Petio this is an ongoing issue on the instability on their servers that are currently being worked on. Despite the behavior, we can confirm that no lost orders were seen and all orders are successfully canceled after the bot's run time.