Missing trades when subscribing to all pairs

[NikiTsv]

Hi, I am trying to listen and gather data for all trades happening on Bittrex. I am using this endpoint: 'https://api.bittrex.com/api/v1.1/public/getmarkets' to get list of all markets (ex. BTC-ETH (eth against btc)) I am subscribing using this code: (pairs being BTC-ETH, BTC-EOS etc)

client.on('trades', function(data){
            eventHandler({data: data.data, pair: data.pair, channel: CHANNELS.Trades});
        });
client.subscribeToMarkets(pairs);

After listening on trades channel for over 20 minutes I have gathered data for like 50 pairs none of which are from the big markets like ETH/BTC, BTC/USDT for example.

I feel like the client is missing a lot of trades. Do you have similar concerns and how can I debug/resolve this?

[aloysius-pgast]

Hi. I usually subscribe to 10-15 pairs only without problem. Maybe too much traffic is generated while subscribing to all pairs. Can you confirm everything is fine when subscribing to USDT-BTC & BTC-ETH only ?

[aloysius-pgast]

When comparing https://global.bittrex.com/Market/Index?MarketName=USDT-BTC with https://mpe-demo.crazyme.net/ui/#/exchanges/bittrex/orderBooks/USDT-BTC (which is using the library internally, I get expected trades. Right now, a few trades seem to happen every 5 min or so, so you can expect a few minutes without any trades events at least for this pair

[NikiTsv]

Hi, @aloysius-pgast , Thanks for the quick reply. Yes I can confirm that it works if I subscribe only for ETH/BTC and BTC/USDT markets.

Do you think it is a viable option to make multiple connections and subscribe for 10-20 pairs each?

[aloysius-pgast]

Market subscriptions generates a lot of traffic because of order books (you can't subscribe to just trades). Using a separate connection for each subset of 10-20 pairs could give you better results. If you still have problem this way (because of too many events on client side), you might need to use separate processes/workers for each subset

[NikiTsv]

I've split all pairs into chunks and I have better results. However there is still missing data for some pairs. I checked BORA/BTC as I have not received any trades for it on this endpoint: https://api.bittrex.com/api/v1.1/public/getmarkethistory?market=BTC-BORA and there seem to be recent trades. Really hard to experiment with the right number of pairs per connection.

[aloysius-pgast]

Thanks for the feedback How many connections did you finally used and how many pairs per connection ? Also did you get timeout events ? Also how long before the most recent trade on BTC-BORA (2020-01-08T10:47:29.06) did you start the client ?

[NikiTsv]

Last time I used 20 pairs per connection. Seems to have missing trades so now I've set them to 10 pairs per connection. Last trade on BTC-BORA was from days ago so I assume there is a cap on subscribtions per connection and the current script ran okay for this period for some pairs but not for BTC-BORA.

Didn't receive any timeout events. Note that I am not sure if I am listening to the right events:

        console.log('connected', wsId)
    })

    client.on('connectionError', ()=>{
        console.log('connectionError', wsId)
    })

    client.on('disconnected', ()=>{
        console.log('disconnected', wsId)
    })```

I'm going to let you know how the new settings are working in a day or two.

[aloysius-pgast]

Yes you're correctly listening to connection-related events. There is also a timeout event which is triggered by the client watchdog when it detects that no market data has been received for more than 30min (across all pairs, not for each pairs) since historically there were some problems where Bittrex stopped sending data without explanation. But after emitting timeout, the client will reconnect so disconnected & connected event would have been emitted also. Let me know how it goes with 10 pairs/con. You're still using a single process, right ? Although I doubt it could be a problem with the event loop being saturated

[NikiTsv]

Since I chunched into 10 pairs per connection 15 hours ago I've received updates for like 300 pairs out of 352. This seems more realistic :)

[aloysius-pgast]

Good news. I'll update the README to advise a limit of 10 pairs per connection next time I publish. Btw, I'm thinking about adding a function in the library to retrieve the list of pairs, so that people don't need to query https://api.bittrex.com/api/v1.1/public/getmarkets manually (since I have the impression that enough people endup retrieving the list of pairs anyway). What do you think ?

[NikiTsv]

@aloysius-pgast Hey, even with 10 pairs per connection after a while (7 months) I concluded that I am not receiving data for all pairs. It seems like there is a rate limit to subscribtions because I tried splitting the traffic both in different websockets and different files (processes).

After tweaking for a few hours this seems to work so far (but I cannot guarantee)

const pairArrays = chunk(pairs, 40)

    for (let i = 0; i < pairArrays.length; i++) {
        const chunkOfPairs = pairArrays[i]
        setTimeout(() => {
            setupWebSocket(chunkOfPairs , handleMessageTrades, handleMessageOrderbooks)
        }, i * 20 * 1000)
    }

What I do here is subscribe to 40 pairs per websocket connection, wait 20 seconds and then subscribe to another chunk of 40 in another websocket connection.

ps. getting all pairs with the library can be useful yes :) Have in mind that the re is a property "IsActive" for each pair on this endpoint "https://api.bittrex.com/api/v1.1/public/getmarkets" and websocket won't return data for inactive markets.

[aloysius-pgast]

I'm in the process of migrating the library to Bittrex v3 api. I hope performances of v3 will be better.