search

jaggedsoft / node-binance-api

Node Binance API is an asynchronous node.js library for the Binance API designed to be easy to use.
MIT License
1.58k stars 767 forks source link

binance.websockets.depthCache fails recently #351

Closed dbvcode closed 4 years ago

dbvcode commented 5 years ago

Hi, I am simply using:

        binance.websockets.depthCache(favMarkets, (symbol, depth) => {
            let bids = binance.sortBids(depth.bids);
            let asks = binance.sortAsks(depth.asks);
            var mdata = {
                bid: Number(binance.first(bids)),
                ask: Number(binance.first(asks)),
                bids: depth.bids,
                asks: depth.asks
            };
            socketService.sendDepth({
                market: symbol,
                depth: mdata
            });
            marketDataService.insertDepth(symbol, mdata);
        });

where favMarkets is about 15 markets. All this worked until half an hour ago. Other WebSocket subscriptions (binance.websockets.trades,binance.websockets.userData,binance.websockets.miniTicker) work.

Now binance.websockets.depthCache fails like this:

Array(1) ["WebSocket closed: 1960720743 (1006)"] node-binance-api.js:46 Array(1) ["WebSocket reconnecting: 1960720743..."] node-binance-api.js:46 Array(1) ["WebSocket closed: 1960720743 (1006)"] node-binance-api.js:46 Array(1) ["WebSocket reconnecting: 1960720743..."] node-binance-api.js:46 Array(1) ["WebSocket closed: 1960720743 (1006)"] node-binance-api.js:46 Array(1) ["WebSocket reconnecting: 1960720743..."] node-binance-api.js:46 Array(1) ["WebSocket closed: 1960720743 (1006)"] node-binance-api.js:46 Array(1) ["WebSocket reconnecting: 1960720743..."] node-binance-api.js:46 Error: IncomingMessage {_readableState: ReadableState, readable: false, _events: Object, …} at api.websockets.depthCache.async.mapLimit (file:///home/john/Testing/node_modules/node-binance-api/node-binance-api.js:1825:37) at (anonymous) (file:///home/john/Testing/node_modules/node-binance-api/node_modules/async/dist/async.js:1140:8) at iteratorSymbol (file:///home/john/Testing/node_modules/node-binance-api/node_modules/async/dist/async.js:473:15) at iterateeCallback (file:///home/john/Testing/node_modules/node-binance-api/node_modules/async/dist/async.js:988:16) at (anonymous) (file:///home/john/Testing/node_modules/node-binance-api/node_modules/async/dist/async.js:969:15) at (anonymous) (file:///home/john/Testing/node_modules/node-binance-api/node_modules/async/dist/async.js:1137:12) at (anonymous) (file:///home/john/Testing/node_modules/node-binance-api/node-binance-api.js:1780:35) at cb (file:///home/john/Testing/node_modules/node-binance-api/node-binance-api.js:109:60) at Request.init.self.callback (file:///home/john/Testing/node_modules/request/request.js:185:21) at emit (events.js:198:12) at (anonymous) (file:///home/john/Testing/node_modules/request/request.js:1161:9) at emit (events.js:198:12) at (anonymous) (file:///home/john/Testing/node_modules/request/request.js:1083:11) at onceWrapper (events.js:286:19) at emit (events.js:203:14) at endReadableNT (_stream_readable.js:1129:11) at _tickCallback (internal/process/next_tick.js:63:18)

What could be the issue here?

dbvcode commented 5 years ago

I dug a bit deeper and found the issue. I think Binance changed some stuff in their API recently and so the problem is: function getSymbolDepthSnapshot calls publicRequest for the depth data. For some symbols, sometimes the response code is 500 with this body:

\r\n500 Internal Server Error\r\n\r\n

500 Internal Server Error

\r\n\r\n\r\n

The first time the error happens, seems to me that it throws it to async.mapLimit from updateSymbolDepthCache function and none of the other symbols get updated and worse the whole app crashes from there and there's no way to treat the error outside the library.

My hack is to make getSymbolDepthSnapshot not return an error but an empty json every time. This prevents the app from crashing and most symbols update. After the first run seems the error is not popping up anymore.

Maybe:

  1. There is a way for the error to be thrown outside the library and not crash the app
  2. There is a way for async.mapLimit to go on after it hits the first error

I'd take care of this and push here, but I have the feeling my solutions are limited as I am no js guru.

ghost commented 4 years ago

I'm getting the same thing but the errors look more like:

[ 'WebSocket closed: xmrbnb@depth (1006)' ]
[ 'WebSocket reconnecting: xmrbnb@depth...' ]
dbvcode commented 4 years ago

It's not the same thing. In my case it crashed the app. I also get the closed and reconnecting messages from time to time as well but this does not crash the nodejs app for me! Another issue with the depth cache is the fact that it first gets a snapshot of the depth over a http call. The problem is Binance will allow only 1200 requests per minute. So if you get the depthCache for a lot of markets and stop and restart your nodejs app a few times in a minute you'll max it out and depthCache will fail! You can check this by reloading a binance page for whatever market you want and if you see the order book is not full of orders, that's the case, your ip has been banned for a while. Hope this helps.

bmino commented 4 years ago

Watch out! The depthCache method, as you noted, will make a REST call to fetch the initial snapshot of each symbol. This call has a variable weight depending on how big the depth cache is.

By default, this size is 500 which corresponds to a weight of 5. You can override this to fetch a depth of 100 which corresponds to a weight of 1 instead.

Use the limit parameter of the websockets.depthCache function like so:

binance.websockets.depthCache(favMarkets, (symbol, depth) => {
    let bids = binance.sortBids(depth.bids);
    let asks = binance.sortAsks(depth.asks);
    var mdata = {
        bid: Number(binance.first(bids)),
        ask: Number(binance.first(asks)),
        bids: depth.bids,
        asks: depth.asks
    };
    socketService.sendDepth({
        market: symbol,
        depth: mdata
    });
    marketDataService.insertDepth(symbol, mdata);
}, 100);  // <----- This additional parameter with a value of 100
dbvcode commented 4 years ago

Thanks for your reply @bmino ! I was aware of the REST calls when I was getting the depth for all markets and found out binance itself was not loading in the browser. :) I was not aware of the depth/weight level being a parameter though. But you mean to say that getting a depth size of 500 will make more calls to binance than a request for a 100 depth size?

jaggedsoft commented 4 years ago

Hi dbvcode, long time no see. I hope you are well.

But you mean to say that getting a depth size of 500 will make more calls to binance than a request for a 100 depth size?

No. It's one call, but a depth size 500 uses 5 times the weight. You are allowed 1200 weight per minute. https://www.binance.com/api/v3/exchangeInfo

If you're connecting to lots of depthCache or chart endpoints at once, you probably need to stagger them with a delay between connections for this reason.

dbvcode commented 4 years ago

Hi @jaggedsoft I'm fine thanks. Hope you are well too. Ok, so that clears it. I thought it was only calls per minute that counts. I guess I can also close this issue. Thanks