WebSocket updates get out of sync with real price

[Ekliptor]

Hi, I am using bitmex-realtime-api 0.4.0

I use websockets to subscribe to the live trade feed for my bot https://wolfbot.org/

this.apiClient = new BitMEXClient({
            testnet: this.apiKey.testnet === true,
            apiKeyID: this.apiKey.key,
            apiKeySecret: this.apiKey.secret,
            maxTableLen: 20000  // the maximum number of table elements to keep in memory (FIFO queue)
        });

I subscribe to trades like this:

            this.apiClient.on('initialize', () => {
                logger.verbose('%s Client initialized, data is flowing.', this.className)

                let lastTradeID = "";

                this.currencyPairs.forEach((pair) => {
                   // loop through currency pairs we want to subscribe to....

                    this.apiClient.addStream(marketPair, 'trade', (data, symbol, tableName) => {
                        // we get maxTableLen newest results, if the table is full it behaves like a FIFO, so we need to detect whats new

                        let newIndex = 0;
                        for (let i=data.length-1; i >= 0; i--) {
                            let trade = data[i];
                            if (trade.trdMatchID == lastTradeID) {
                                newIndex = i+1;
                                break;
                            }
                        }
                        lastTradeID = data[data.length-1].trdMatchID;

                        let trades = [];
                        let timestamp = Math.floor(Date.now() / 1000);
                        for (let i=newIndex; i < data.length; i++) {
                            let trade = data[i];
                       }
               }
        });

The trade-price already start with 2-3 seconds latency and after 30+ minutes running the bot, the prices are clearly wrong. Often they are off bei 50 - 100 USD (!) for the XBTUSD contract.

Is this a known issue with your API server? Can you point me to a solution where I can get the correct prices with very low latency? Thx I am not using testnet, using mainnet.

[nhondong]

Hi, I even have the same problem. just working arround using another IP, but I dont know how long this will last...

[Ekliptor]

For me this problem isn't IP specific. Happening with a German home IP as well as various server IPs. Where is your IP located?

[nhondong]

Hi, the IP is located in Germany ;) It is a business IP from Vodafone and another IP from Netcologne Even a Telekom IP get the same results.

So it has something to do with the Python Code.

[Ekliptor]

This is nodejs code. Which python code are you using that sometimes receives full updates? A shame if bitmex really doesn't want to fix their nodejs API...

[nhondong]

Sorry, I am using the python websocket code. I wanna try the delta-server now, but polling is not really professional.

But if the delta-server uses the same code, I think, i will get the same results. :(

Maybe doing a reconnect will work arround it

[bendelo]

Make sure you are comparing the prices against testnet.bitmex.com

[Ekliptor]

On testnet there are fewer trades, so it's harder to see that anything is missing. But even there are trades missing. And the longer your delta server of bitmex-realtime-api the more it gets out of sync. I have seen other devs on telegram with the same problem.

What's the recommended fix? Not using delta server at all? HTTP polling everything all the time? (causing more server load while trying to avoid bans) Why has this been closed?

[nhondong]

Closed? Are you kidding me???

[nhondong]

The problem exists on the Mainnet with python and nodejs. If you are running a service getting trades, orderbooks and ticker, it gets out of sync and did not come back until a restart. So restarting the service is a workarround, but not a realy solution.

[Ekliptor]

bump. any updates?

[nhondong]

@Ekliptor: I had a memory leak, which causes a very high delay. but this was just one problem. currently I check the timestamp of the incoming trade. If there are too many with high delays, i restart the websocket connection. But this is not a nice solution. Because in case of a pump oder dump there are often high delays...

[datourezi]

@Ekliptor: I had a memory leak, which causes a very high delay. but this was just one problem. currently I check the timestamp of the incoming trade. If there are too many with high delays, i restart the websocket connection. But this is not a nice solution. Because in case of a pump oder dump there are often high delays...

when i restart the websocket connection，Still high latency，even i use other Proxy ip what happened? it can distinguish Proxy ip，Lock your machine？

[mjzarrin]

for me, the latest price in the following python script after 24 hours is 11400$ however the real price is about 12500$. any thoughts?

`def now_ms(): return int(round(time.time() * 1000))

def run(): ws = BitMEXWebsocket(endpoint="wss://www.bitmex.com/realtime/websocket", symbol="XBTUSD") while(ws.ws.sock.connected): try: tick = {'timestamp': now_ms(),'price':ws.get_ticker()['last']} print('inserted into db',tick) time.sleep((1000 + tick['timestamp'] - now_ms())/1000) except Exception as e: print(e) run()`

[LooOOooM]

same problem here. Any progress on this issue?

[andersea]

Hmmm... I have been running a websocket feed for weeks on my own trading dashboards. It is always in sync. Here is an example:

The dark blue insert is the price from the live website, it is the tab title on the browser. The chart shows my live feed from the socket. The difference you are seeing on this is just a rounding error in the chart and slight lag on the chart updating, the backend data is always up to date from what I am seeing.

So, not sure what is going on for you guys, but it seems to work for me.

What I would probably do, if I were you, is to go directly to the websocket consumer and insert a log point and see what the data looks like as it comes in. In the python bot this is easy. Just set log level to debug, there is already a log message in the _on_message function.

All messages include a timestamp, so it should be easy to see if there is lag on the feed.

[surGeonGG]

I'm having the same problem with the bitmex-ws python library.

Did anyone manage to fix this yet?

[Philipp-Chvetsov]

So is there a fix for this in Python?

[Philipp-Chvetsov]

So is there a fix for this in Python?

Seems to be solved by using market_depth() with orderBook10.