Bitmex testnet. Errors handle_orders

ShieldTrade commented 1 year ago

Operating System

windows 10

Programming Languages

python

CCXT Version

2.2.93

Description

Using bitmex testnet for 'symbol': 'BTC/USD:BTC' I get this error when trying to amend pr.

Exception in callback FastClient.receive_loop.<locals>.handler() at C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\base\fast_client.py:20
handle: <Handle FastClient.receive_loop.<locals>.handler() at C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\base\fast_client.py:20>
Traceback (most recent call last):
  File "C:\Users\lucio\AppData\Local\Programs\Python\Python39\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\base\fast_client.py", line 25, in handler
    self.handle_message(message)
  File "C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\base\aiohttp_client.py", line 32, in handle_message
    self.handle_text_or_binary_message(message.data)
  File "C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\base\aiohttp_client.py", line 27, in handle_text_or_binary_message
    self.on_message_callback(self, decoded)
  File "C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\bitmex.py", line 1277, in handle_message
    return method(client, message)
  File "C:\ShieldTrade\Python_projects\venv\lib\site-packages\ccxt\pro\bitmex.py", line 792, in handle_orders
    client.resolve(self.orders, messageHash + ':' + symbol)
TypeError: can only concatenate str (not "NoneType") to str

when error happens CCXT variables

self.orders

[{'info': {'orderID': '2fe690a7-3c33-46fa-8b6c-e283fea47bbb', 'account': 376818, 'symbol': 'XBTUSD', 'side': 'Sell', 'orderQty': 100, 'price': 24467.0, 'currency': 'USD', 'settlCurrency': 'XBt', 'ordType': 'Limit', 'timeInForce': 'GoodTillCancel', 'execInst': 'ParticipateDoNotInitiate', 'ordStatus': 'New', 'workingIndicator': True, 'leavesQty': 100, 'cumQty': 0, 'text': 'Submitted via API.', 'transactTime': '2023-02-17T17:23:21.264Z', 'timestamp': '2023-02-17T17:23:21.264Z'}, 'id': '2fe690a7-3c33-46fa-8b6c-e283fea47bbb', 'clientOrderId': None, 'timestamp': 1676654601264, 'datetime': '2023-02-17T17:23:21.264Z', 'lastTradeTimestamp': 1676654601264, 'symbol': 'BTC/USD:BTC', 'type': 'limit', 'timeInForce': 'GTC', 'postOnly': True, 'side': 'sell', 'price': 24467.0, 'stopPrice': None, 'amount': 100.0, 'cost': 0.0, 'average': None, 'filled': 0.0, 'remaining': 100.0, 'status': 'open', 'fee': None, 'trades': [], 'fees': []}, {'info': {'orderID': 'db9358b3-3443-4f42-8be2-a70f99d0ddf9', 'account': 376818, 'symbol': 'XBTUSD', 'side': 'Buy', 'orderQty': 100, 'price': 24288.0, 'currency': 'USD', 'settlCurrency': 'XBt', 'ordType': 'Limit', 'timeInForce': 'GoodTillCancel', 'execInst': 'ParticipateDoNotInitiate', 'ordStatus': 'New', 'workingIndicator': True, 'leavesQty': 100, 'cumQty': 0, 'text': 'Submitted via API.', 'transactTime': '2023-02-17T17:23:21.298Z', 'timestamp': '2023-02-17T17:23:21.298Z'}, 'id': 'db9358b3-3443-4f42-8be2-a70f99d0ddf9', 'clientOrderId': None, 'timestamp': 1676654601298, 'datetime': '2023-02-17T17:23:21.298Z', 'lastTradeTimestamp': 1676654601298, 'symbol': 'BTC/USD:BTC', 'type': 'limit', 'timeInForce': 'GTC', 'postOnly': True, 'side': 'buy', 'price': 24288.0, 'stopPrice': None, 'amount': 100.0, 'cost': 0.0, 'average': None, 'filled': 0.0, 'remaining': 100.0, 'status': 'open', 'fee': None, 'trades': [], 'fees': []}, {'info': {'orderID': 'db9358b3-3443-4f42-8be2-a70f99d0ddf9', 'account': 376818, 'price': 21915.5, 'text': 'Amended price: Amended via API.\nSubmitted via API.', 'transactTime': '2023-02-17T17:24:47.921Z', 'timestamp': '2023-02-17T17:24:47.921Z'}, 'id': 'db9358b3-3443-4f42-8be2-a70f99d0ddf9', 'clientOrderId': None, 'timestamp': 1676654687921, 'datetime': '2023-02-17T17:24:47.921Z', 'lastTradeTimestamp': 1676654687921, 'symbol': None, 'type': None, 'timeInForce': None, 'postOnly': None, 'side': None, 'price': 21915.5, 'stopPrice': None, 'amount': None, 'cost': None, 'average': None, 'filled': None, 'remaining': None, 'status': None, 'fee': None, 'trades': [], 'fees': []}]

messageHash ` order `

symbol None

For some reason I cannot understand symbol changes to None is causing the error

Code

carlosmiei commented 1 year ago

Hello @ShieldTrade, What do you mean by "trying to amend pr."?

ShieldTrade commented 1 year ago

sending an amend price order

carlosmiei commented 1 year ago

@ShieldTrade But you got the error using watchOrders, right?

carlosmiei commented 1 year ago

@ShieldTrade I found a small authentication issue that will be fixed shortly, but I can't reproduce the issue reported:

Python v3.10.9
CCXT v2.8.14
bitmex.watchOrders(BTC/USD:BTC)
[{'amount': 200.0,
  'average': None,
  'clientOrderId': None,
  'cost': 0.0,
  'datetime': '2023-02-18T14:25:10.405Z',
  'fee': None,
  'fees': [],
  'filled': 0.0,
  'id': 'a53c28f0-4088-4360-a27c-7c332c771d2d',
  'info': {'account': 395724,
           'cumQty': 0,
           'currency': 'USD',
           'leavesQty': 200,
           'ordStatus': 'New',
           'ordType': 'Limit',
           'orderID': 'a53c28f0-4088-4360-a27c-7c332c771d2d',
           'orderQty': 200,
           'price': 11000,
           'settlCurrency': 'XBt',
           'side': 'Buy',
           'stopPx': None,
           'symbol': 'XBTUSD',
           'text': 'Amended orderQty price: CCXT\nCCXT',
           'timeInForce': 'GoodTillCancel',
           'timestamp': '2023-02-18T14:25:10.405Z',
           'transactTime': '2023-02-18T14:25:10.405Z',
           'workingIndicator': True},
  'lastTradeTimestamp': 1676730310405,
  'postOnly': False,

ShieldTrade commented 1 year ago

Hi @carlosmiei ,

Steps to reproduce the issue:

1) place the following code in line 767 of CCXT_pro bitmex.py. This will print the message received ( before handle ) and hopefully will give you a starting point to investigate the issue. print ('ccxt_message', data)

2 Run the following snippet code that will place a limit order at level 3 bid price after 10 sec. After 20 second the same order will be amended ( new price will be level 5 bid). I have the same CCXT error when watching orders and manually amend price ( from web )

import ccxt.pro
import ccxt.async_support
import asyncio

order_id = None
lmt_pr = 0  # set to bid 3 level every time a book is received
amnd_pr = 0  # set to bid 5 level every time a book is received

async def bitmex_test():
    bitmex_test = ccxt.pro.bitmex({'verbose': False,
                                  'enableRateLimit': False,
                                  'apiKey': '5TsTycPxxxxxxxxxxxxx79jwTiw',
                                  'secret': 'o-Q2LEHJgTWxxxxxxxxGJyKROc7yMKvVZnv10Gj6',
                                  'newUpdates': True,
                                  'options': {
                                      'tradesLimit': 1000,
                                      'OHLCVLimit': 1000,
                                      'ordersLimit': 1000,
                                  },
                                  })
    if bitmex_test.urls['test']:  # mi sposto sulla sandbox
        bitmex_test.urls['api'] = bitmex_test.urls['test']
    markets = await bitmex_test.load_markets()  # carica tutte le info relative al mercato. è possibile stamparle
    print("bitmex_test loaded")
    await bitmex_test.sleep(3000)
    return bitmex_test

async def ccxt_watch_order(exchange, symbol):
    print("ccxt_watch_orders", exchange.id, symbol)
    global order_id
    while True:
        try:
            order = await exchange.watch_orders(symbol)
            print()
            print('order', order)
            order_id = order[0]['id']
        except Exception as e:
            print('Exception', e)

async def ccxt_watch_book(exchange, symbol, limit):
    global lmt_pr
    global amnd_pr
    print ('ccxt_watch_book:', exchange.id, symbol, limit)
    while True:
        try:
            book = await exchange.watch_order_book(symbol, limit)
            print('book ', book['bids'][0][0], ' ', book['asks'][0][0])
            lmt_pr = book['bids'][2][0]
            amnd_pr = book['bids'][4][0]
        except Exception as e:
            print('Exception', e)

async def on_time_send_order(exchange, symbol, mls = 1000):
    cont = 0
    while True:
        cont = cont + 1
        if cont == 10:  # 10'  send lmt ord
            params = {'execInst': 'ParticipateDoNotInitiate'}
            await exchange.create_order(symbol, 'limit', 'buy', 100, lmt_pr, params= params)
        if cont == 20: # 10'  send amend_pr ord
            await exchange.private_put_order({"orderID": order_id, "price": amnd_pr})
        print ('time ', cont)
        await asyncio.sleep(mls/1000)

async def main():
    exchange = await bitmex_test()
    symbol = 'BTC/USD:BTC'
    loops = [
        ccxt_watch_order(exchange, symbol),
        ccxt_watch_book(exchange, symbol, None),
        on_time_send_order(exchange, symbol)
    ]
    await asyncio.gather(*loops)

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

asyncio.run(main())

ShieldTrade commented 1 year ago

Hi @carlosmiei, After further investigation I can tell you the problem is only for the testnet.

Testnet wrong message ccxt_message [{'orderID': 'de12b561-7f00-458e-86ec-00882f9c661d', 'account': 376818, 'price': 24599.5, 'text': 'Amended price: Amend from testnet.bitmex.com\nSubmission from testnet.bitmex.com', 'transactTime': '2023-02-19T15:07:29.265Z', 'timestamp': '2023-02-19T15:07:29.265Z'}]

Mainnet Correct message ccxt_message [{'orderID': '5a213cf9-5f5c-481b-b82f-772a0383671a', 'price': 23999.5, 'text': 'Amended price: Amend from www.bitmex.com\nSubmission from www.bitmex.com', 'transactTime': '2023-02-19T15:11:04.729Z', 'timestamp': '2023-02-19T15:11:04.729Z', 'clOrdID': '', 'account': 1644444, 'symbol': 'XBTUSD'}]

The testnet message does not have the'symbol': 'XBTUSD' probably causing the problem.

In mainnet sending an amend price order works correctly. It is an issue depending on bitmex. We can consider this closed.

Thank you for your time.

carlosmiei commented 1 year ago

ccxt_message [{'orderID': 'de12b561-7f00-458e-86ec-00882f9c661d', 'account': 376818, 'price': 24599.5, 'text': 'Amended price: Amend from testnet.bitmex.com\nSubmission from testnet.bitmex.com', 'transactTime': '2023-02-19T15:07:29.265Z', 'timestamp': '2023-02-19T15:07:29.265Z'}]

@ShieldTrade thank you for helping me debug it, we will escalate this issue to Bitmex.

ShieldTrade commented 1 year ago

@carlosmiei . Just to let you know Bitmex has a problem with orderbook on testnet as well. You can simply verify it by printing the first 3/4 orderbook levels and manually insert and cancel a few orders. You will see the book on the web is different from the one printed by CCXT. Mainnet works correctly. I opened a ticket with bitmex support myself. I guess your ticket have highest priority 😊.

ShieldTrade commented 1 year ago

Hi @carlosmiei , I just got an answer from bitmex

I have just tested this websocket subscription on Testnet and have not found the problem you describe.

At this stage I am not sure the ccxt_message is the raw data received by Bitmex or it has already been modified by CCXT.

place the following code in line 767 of CCXT_pro bitmex.py. This will print the message received ( before handle ) and hopefully will give you a starting point to investigate the issue. print ('ccxt_message', data)

In any case there is no problem in Mainnet. Please let me know your opinion on this issue

thank you

ShieldTrade commented 1 year ago

hi @carlosmiei, could you reproduce the issue?

ccxt / ccxt