Failed Test PR #294: Failed to fetch status for an order

[gerald-hb]

Failed to fetch status for an order and becomes stuck in the exchange (9 hours). I have checked the logs and it seems that this order gets filled. Affected order ID: SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec

2023-01-30 19:39:18,206 - 1101087 - hummingbot.connector.gateway.clob_spot.gateway_clob_spot.GatewayCLOBSPOT - WARNING - Failed to fetch status update for the order SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec.
2023-01-30 19:39:27,033 - 1101087 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (INJ-USDT) Canceling the limit order SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec. [clock=2023-01-30 11:39:27+00:00]
2023-01-30 19:40:00,471 - 1101087 - hummingbot.connector.client_order_tracker - INFO - The SELL order SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec amounting to 0.200000000000000000/0.200000000000000000 INJ has been filled.
2023-01-30 19:40:00,472 - 1101087 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (INJ-USDT) Maker sell order of 0.200000000000000000 INJ filled. [clock=2023-01-30 11:40:00+00:00]
2023-01-30 19:40:00,504 - 1101087 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1675078800.0, "order_id": "SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec", "trading_pair": "INJ-USDT", "trade_type": "TradeType.SELL", "order_type": "OrderType.LIMIT_MAKER", "price": "2.767", "amount": "0.200000000000000000", "trade_fee": {"percent": "0", "percent_token": null, "flat_fees": [{"token": "USDT", "amount": "-0.00005534"}]}, "exchange_trade_id": "25107274_22_0", "exchange_order_id": "0x7552b2a5dff821a03bafe21b59936f3f3da518c239f1838a96aa74ba9ed84d17", "leverage": 1, "position": "NIL", "event_name": "OrderFilledEvent", "event_source": "injective_injective_mainnet"}
2023-01-30 19:40:00,505 - 1101087 - hummingbot.connector.client_order_tracker - INFO - The SELL order SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec amounting to 0.400000000000000000/0.200000000000000000 INJ has been filled.
2023-01-30 19:40:00,505 - 1101087 - hummingbot.strategy.pure_market_making.pure_market_making - INFO - (INJ-USDT) Maker sell order of 0.200000000000000000 INJ filled. [clock=2023-01-30 11:40:00+00:00]
2023-01-30 19:40:00,531 - 1101087 - hummingbot.core.event.event_reporter - EVENT_LOG - {"timestamp": 1675078800.0, "order_id": "SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec", "trading_pair": "INJ-USDT", "trade_type": "TradeType.SELL", "order_type": "OrderType.LIMIT_MAKER", "price": "2.768", "amount": "0.200000000000000000", "trade_fee": {"percent": "0", "percent_token": null, "flat_fees": [{"token": "USDT", "amount": "-0.00005536"}]}, "exchange_trade_id": "25107274_22_1", "exchange_order_id": "0x9b9d288efa11ce55238635a4fa61356e76747d28656931dc83adf7755f9d80e6", "leverage": 1, "position": "NIL", "event_name": "OrderFilledEvent", "event_source": "injective_injective_mainnet"}
2023-01-30 19:40:09,123 - 1101087 - hummingbot.connector.gateway.clob_spot.gateway_clob_spot.GatewayCLOBSPOT - NETWORK - Error fetching status update for the order SIJUT5f3799c64de2879a242afa40647219bfee14caa9bf0ec: <AioRpcError of RPC that terminated with:
    status = StatusCode.INVALID_ARGUMENT
    details = "invalid hash"
    debug_error_string = "UNKNOWN:Error received from peer ipv4:34.138.231.254:443 {created_time:"2023-01-30T19:40:09.118604456+08:00", grpc_status:3, grpc_message:"invalid hash"}"

Steps to reproduce:

1. Clone and install feature branch
2. configure gateway
3. connect injective wallet
4. create a simple pure mm strategy
5. Run the bot long term

Attachments: inj-packets.zip

Wallet use: 0xE973154ABAA89CCBc0AF5Ff3d4dca63A21EAd958 / inj1a9e32j464zwvhs90tleafh9x8gs74k2cfda0xy

[gerald-hb]

Running now for 13 hours. So far everything is ok. Attaching logs here for review inj-pmm.zip

[RC-13]

Closing ticket, issue is confirmed a server side problem. PR has been submitted as well to foundation.