Python websocket doesn't get the right live data

[2803media]

Hi I use the websocket with a python 3.6 script and I notice that after some couple of hours of running the data is not the real live value for example right now the live BTC is 7897 and I get 7922 with websocket (the script was launched 12 hours ago).

Here is my script:

from bitmex_websocket import BitMEXWebsocket
import logging
from logging.handlers import RotatingFileHandler
from time import sleep
import datetime
import json

crypto = "XBTUSD"
api_key = "XXXXX"
api_secret = "YYYYY"
sleeping = 5

def run():
    logger = setup_logger()

    ws = BitMEXWebsocket(endpoint="https://www.bitmex.com/api/v1",
                         symbol=crypto, api_key=api_key, api_secret=api_secret)
    logger.info("Instrument data: %s" % ws.get_instrument())
    while(ws.ws.sock.connected):
        logger.info("Ticker: %s" % ws.get_ticker())
        sleep(sleeping)

def setup_logger():
    # Prints logger info to terminal
    logger = logging.getLogger()
    # Change this to DEBUG if you want a lot more info
    logger.setLevel(logging.INFO)
    ch = logging.StreamHandler()
    # create formatter
    formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    file_handler = RotatingFileHandler(
        '/home/test/public_html/block/bot/_activity_' + api_key + '.log', 'a', 1000000, 1)
    file_handler.setLevel(logging.DEBUG)
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    # add formatter to ch
    ch.setFormatter(formatter)
    logger.addHandler(ch)
    return logger

if __name__ == "__main__":
    run()

Thanks for your feedback on this lag or bug !

[mellertson]

@k0rean-rand0m Thanks for posting that code! It's definitely very helpful! Will private data like orders, fills, and positions be secure if transferred from bitmex to a wss client using code like you suggested? I assume it would be, due to the URI being "wws://*", but just wanted to confirm. I suppose I could always confirm using a network sniffer. But, figured it might be quicker to ask. Thanks again!

P.S. A couple years ago I had issues connecting to HitBTC with the same symptoms described in this issue. I re-wrote the web-socket client a few times trying to fix it and eventually gave up. But, I do remember now that using the websockets library with asyncio worked without interruption. I had forgotten all about the websockets library. So cool that half a world a way you were there to point me in the right direction! Cheers mate!

[stabilus]

The delta-server also produces lags (not just the python websocket). I did quite a bit of testing and found these results:

Lag duration varies

• with the volume of information it has to load from bitmex
• with the strain I put on the delta server by polling it

I created an endless loop polling the trade data (http://localhost:4444/trade?), selecting the last 10 seconds of data and calculating the volumes (in panda dataframes). Not applying any pause to the loop, I got about 10 polls/second. This was manageable during low volume periods. Comparing the timestamp of the latest trade with the UTC timestamp I got a delay of typically less than 1 second.

However during periods of higher volumes (number of trades / second > 20) the lag time started to increase - the higher the number of trades, the faster the increase of the lag. Once the number of transactions decreased to below ca. 20/s, the lag started to recover and ended up back under a second.

Then I built a pause (say 0.1 s) into the loop. That yielded ca 5 polls / s. I found that there was far less lag. This loop could handle approx. 100 trades/s without building lag. It would increase to 2 seconds but quickly go back to below 1 s.

Also, straining the delta server with polls from a different app that ran in parallel influenced the lag negatively.

To conclude: The delta server needs "room to breath". It needs some processing power to load the data from bitmex. If you overload your server with too many polls it falls behind and you get a lag that increases in response to the data volume it needs to handle.

So, I ended up using a lag measurer and dynamically adapting my polling rate to the lag. Works for me.

[globophobe]

BitMEX has a REST API, which is nice.

If the client disconnects, wait for reconnect. Buffer the incoming data, while making a request to the REST API. Finally, filter the buffered data for missing trdMatchID from the REST API.

[janisdombr]

I try to collect 'orderBookL2' data but while every 10 seconds it's about 5-10% of messages with error because can't find id with that price. It occurs because many messages with action "delete" came before "update" with the same id. I thought that it's because of missing "insert" and I use your approach with asyncio lib but then I realised that if I ignore "delete" messages all is ok. But messages of 'orderBookL2' have no attribute 'timestamp'. Maybe you know how to solve it? I can update 'size' to 0 instead deleting row but it can be wrong way...

Yep, the error is still here, but for those, who want to use Python + WebSocket and have have face with this awful bug, here's something, that you can do:

import websockets
import asyncio
import json

async def capture_data():
    uri = "wss://www.bitmex.com/realtime?subscribe=instrument:XBTUSD"
    async with websockets.connect(uri) as websocket:
    while True:
        data = await websocket.recv()
        data = json.loads(data)
        print('\n', data)

asyncio.get_event_loop().run_until_complete(capture_data())

Docs: https://www.bitmex.com/app/wsAPI https://websockets.readthedocs.io/en/stable/

I've checked it and data was identical with the BitMEX web-interface. No delays at all.

[mandri87]

Yep, the error is still here, but for those, who want to use Python + WebSocket and have have face with this awful bug, here's something, that you can do:

import websockets
import asyncio
import json

async def capture_data():
    uri = "wss://www.bitmex.com/realtime?subscribe=instrument:XBTUSD"
    async with websockets.connect(uri) as websocket:
    while True:
        data = await websocket.recv()
        data = json.loads(data)
        print('\n', data)

asyncio.get_event_loop().run_until_complete(capture_data())

Docs: https://www.bitmex.com/app/wsAPI https://websockets.readthedocs.io/en/stable/

I've checked it and data was identical with the BitMEX web-interface. No delays at all.

I'have the same issue, data are delayed and I need a solution to get it in real time. I want to implement your proposal but I can't understand where I should put this piece of code in bitmex_websocket.py Could you or someone else please help me? Thanks in advance

[globophobe]

Much recommended https://github.com/bmoscon/cryptofeed

[tuncel3]

I have same problem with binance websocket aggregate trades stream. When market volatility increase, time stamp of websocket received data deviates from actual time.

I think it is because of client computer CPU speed. When there are many trades happening in volatile times, it becomes harder for client computer to handle so many "on message" interrupts and it queues them. After queue finishes, for example market volatility is reduced, it becomes normal.

I think the solution is to use a faster CPU on client side. Someone said he uses threading to receive data, if CPU doesn't have extra threads, python extra threads will not behave like extra threads.