Dexalot - Unable to submit orders followed by error fetching status update for the active order

[rapcmia]

Describe the bug When checking dexalot connector, observed that on the client it shows an active order for few seconds and eventually removed followed by an Error fetching status update for the active order.... Also while it is true that there is active order on status, the log panel does not indicate a successful created order and no order available on exchange

https://github.com/hummingbot/gateway/assets/73840223/3a8e09a9-7ca8-4d88-90f3-96bb1269f9a9

On the recording above:

• Observed that it displayed an order yet no "created successfully" on log panel
• Active order on the status would eventually disappear after few minutes and followed with an error:

2023-08-16 11:16:10,135 - 15 - hummingbot.connector.gateway.clob_spot.gateway_clob_spot.GatewayCLOBSPOT - WARNING - Error fetching status update for the active order 0x0000000000000000001a8ecc747aee4b313d4fcd504eef4c7d7311e204a5923a: 'NoneType' object has no attribute 'get'.
2023-08-16 11:16:16,681 - 15 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1692184576.0, "order_id": "0x0000000000000000001a8ecc747aee4b313d4fcd504eef4c7d7311e204a5923a", "order_type": "OrderType.LIMIT_MAKER", "event_name": "MarketOrderFailureEvent", "event_source": "dexalot_avalanche_dexalot"}
2023-08-16 11:16:16,681 - 15 - hummingbot.connector.client_order_tracker - INFO - Order 0x0000000000000000001a8ecc747aee4b313d4fcd504eef4c7d7311e204a5923a has failed. Order Update: OrderUpdate(trading_pair='AVAX-USDC', update_timestamp=1692184576.6427057, new_state=<OrderState.FAILED: 6>, client_order_id='0x0000000000000000001a8ecc747aee4b313d4fcd504eef4c7d7311e204a5923a', exchange_order_id=None, misc_updates=None)
2023-08-16 11:16:24,818 - 15 - hummingbot.client.hummingbot_application - INFO - stop command initiated.

Ran for few more hours and no luck on getting an order submitted to the exchange, account has enough balance to create and running on a cloud server

Steps To Reproduce

1. Setup dexalot on PMM
2. Observe behavior

Release version dev-1.19.0

Attachments logs_pmm-dexalot-02.log pmm-dexalot-02.yml.log remove the .log

[nikspz]

Upd: this issue happened for the CLOB connectors that using gateway (XRPL etc.)

Had some failture events on the order creation/refreshing but its actually different one logs_conf_pure_mm_1.log

[rapcmia]

Test update:

• Setup docker compose build on cloud server
• Test on latest development client and gateway

---

Recorded a dexalot connector behavior while crosshecking with exchange https://drive.google.com/file/d/1ZtNvYKduHXN1NkU35KuTLiYaiAT2xVuO/view?usp=sharing file size too large 😓

• Observe client trying to create orders

  • 00:00:37s
    • The client tries to create orders however its on ask order that was recognized. Exchange shows both bid and ask
    • It is observed that there are Failed to fetch trade updates... for bid order
    • This cause the order to be stuck on exchange 0x0000000000000000001dfd846dc347698bb165cc223fc2180d1ba929e0f8a248
    • When order refresh time is reached, ask order is cancelled successfully and bid order not cancelled ❌

  • 00:01:22s

    • On the second order cycle observed that the connector tries again to create orders
    • This time getting errors related to exchange order id Failed to fetch trade updates for order...Error: No update found for order...

    • Observed that on the screenrecording, it has 3 orders {buy, sell, sell}

      2023-09-15 11:05:21,131 - 16 - hummingbot.connector.gateway.clob_spot.gateway_clob_spot.GatewayCLOBSPOT - WARNING - Failed to fetch trade updates for order 0x0000000000000000001dfd846dc347698bb165cc223fc2180d1ba929e526bbe6. Error: No update found for order 0x0000000000000000000000000000000000000000000000000000000064b3be4f.
      Traceback (most recent call last):
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/dexalot/dexalot_api_data_source.py", line 454, in _get_order_status_update_with_order_id
        resp = await self._api_get(
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/dexalot/dexalot_api_data_source.py", line 532, in _api_get
        return await self._api_request(*args, **kwargs)
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/dexalot/dexalot_api_data_source.py", line 558, in _api_request
        request_result = await rest_assistant.execute_request(
      File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 47, in execute_request
        response = await self.execute_request_and_get_response(
      File "/home/hummingbot/hummingbot/core/web_assistant/rest_assistant.py", line 99, in execute_request_and_get_response
        raise IOError(f"Error executing request {method.name} {url}. HTTP status is {response.status}. "
      OSError: Error executing request GET https://api.dexalot.com/privapi/signed/orders/0x0000000000000000000000000000000000000000000000000000000064b3be4f. HTTP status is 404. Error: {"message":""}
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
      File "/home/hummingbot/hummingbot/connector/exchange_py_base.py", line 954, in _update_orders_fills
        trade_updates = await self._all_trade_updates_for_order(order=order)
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/gateway_clob_spot.py", line 599, in _all_trade_updates_for_order
        trade_updates = await self._api_data_source.get_all_order_fills(in_flight_order=order)
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/dexalot/dexalot_api_data_source.py", line 192, in get_all_order_fills
        await self.get_order_status_update(in_flight_order=in_flight_order)
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/dexalot/dexalot_api_data_source.py", line 180, in get_order_status_update
        status_update = await self._get_order_status_update_with_order_id(in_flight_order=in_flight_order)
      File "/home/hummingbot/hummingbot/connector/gateway/clob_spot/data_sources/dexalot/dexalot_api_data_source.py", line 459, in _get_order_status_update_with_order_id
        raise ValueError(f"No update found for order {in_flight_order.exchange_order_id}.")
      ValueError: No update found for order 0x0000000000000000000000000000000000000000000000000000000064b3be4f.

Steps to reproduce:

1. Setup PMM with dexalot
2. Observe behavior on creating and cancellation of order

logs_pmm-dexalot-01.log pmm-dexalot-01.yml.log remove .log to use logs_gateway_app.log.2023-09-15.log

[petioptrv]

@rapcmia, it sounds like the issue description is different from the follow-up. In the original description, it looks like the order did actually fail also on the Dexalot exchange itself, but the bot, although it logged the failure, continued to show the order as active in the order status update. Is that correct?

In your follow-up, the situation is that the bot thought the order had failed, but it actually was successful on the exchange. However, the provided client logs and gateway logs do not overlap... So I am not sure what happens on gateway when the client labels the order as failed at 11:04:10. Also, I don't have the client logs that correspond to the failure observed in the gateway log

[rapcmia]

Hi @petioptrv thanks for the reply We are not able to reproduce the issue on orders that are active on client yet failed on dexalot exchange. However we still observe the behavior where the order had failed on client but successful on exchange resulting the order to be not recognized. Created separate ticket instead https://github.com/hummingbot/gateway/issues/209