Long Running Price Retrieval

[mattgi]

Hi,

First, great library! I've been waiting to see something for nodejs for months. Awesome job! I am looking forward to the new BF REST based API but, until that comes, I'm loving your module.

One thing I need an example on...

Typically I use a queue based approach whereby I can throw an active Market on to a queue and have a process pick it off the queue, get prices for that market, save those prices to a DB, and then add it back on the queue to be processed again. This lets me throttle the process picking messages off the queue based on the task at hand (e.g. I can limit GetMarket messages to 5 calls per min, or GetPrices to 60 calls, etc) and it works quite well. The use-case is just getting prices for, say, all Match Odd markets for NBA games.. there might be 10 games on at the same time, so I need to get prices for them all and manage things concurrently.

What I'm wondering is how to manage the session in all of this.

In .NET I use a static dictionary and keep about 20 sessions alive.. so when a request comes to get prices, I grab the first available session token and use that.. this saves me from logging in and out for every request.. I just remove the session token from the dictionary and then add the new token back in when the response comes back. When no sessions exist, it just logs in and gets one. This means I am not continually logging in or keeping things alive. It is also self contained within my bf service code so, from a consumer, it just says 'get prices' and, within the get prices method, it checks to see if a session exists and, if it doesn't, forces the log in - nicely abstracted away.

Based on how you have structured your module, are you able to provide an example on how you would approach such a scenario with respect to Login/session. For example, how would you get prices for 10 markets every minute? I have just started to use Kue for the queuing and it is working nicely but am really interested in how you would approach it.

Thanks

[AlgoTrader]

First, great library! I've been waiting to see something for nodejs for months. Awesome job! I am looking forward to the new BF REST based API but, until that comes, I'm loving your module.

As soon as a new API is there, there will be a complete rewrite available.

Typically I use a queue based approach whereby I can throw an active Market on to a queue and have a process pick it off the queue, get prices for that market, save those prices to a DB, and then add it back on the queue to be processed again. This lets me throttle the process picking messages off the queue based on the task at hand (e.g. I can limit GetMarket messages to 5 calls per min, or GetPrices to 60 calls, etc) and it works quite well. The use-case is just getting prices for, say, all Match Odd markets for NBA games.. there might be 10 games on at the same time, so I need to get prices for them all and manage things concurrently.

That's not a node way. In node, you can write to DB and process in parallel. node.js never blocks.

In .NET I use a static dictionary and keep about 20 sessions alive.. so when a request comes to get prices, I grab the first available session token and use that.. this saves me from logging in and out for every request.. I just remove the session token from the dictionary and then add the new token back in when the response comes back. When no sessions exist, it just logs in and gets one. This means I am not continually logging in or keeping things alive. It is also self contained within my bf service code so, from a consumer, it just says 'get prices' and, within the get prices method, it checks to see if a session exists and, if it doesn't, forces the log in - nicely abstracted away.

What you mean with sessions? betfair-sports-api may open 30 simultaneous connections to BF, you never cares of transport HTTPS connections. But the session is single. Nobody prevent you from opening multiple sessions. I see no reasons for having multiple sessions for other reasons that overcoming API throttling. When you need getPrices, you just ask for prices, a free HTTPS connection is found automatically. But the sessionToken is the same for all the HTTPS connections.

Based on how you have structured your module, are you able to provide an example on how you would approach such a scenario with respect to Login/session. For example, how would you get prices for 10 markets every minute? I have just started to use Kue for the queuing and it is working nicely but am really interested in how you would approach it.

Just do not care. I use FreeAPI, so every second a single market gets a right to refresh. If market is important it is refreshed often, if not - not so often. The only problem is throttling. getMarketPricesCompressed is 60p/m so any app that calls it once a second is valid.

[mattgi]

Thanks for your reply. I was referring to the BF sessionToken, not HTTPS but I think I'm on the same page based on your response either way.

I know everything should be (and is) non-blocking but I am interested in how you handle throttling the calls made upon the API. For example, do you just ensure you wait (e.g. setTimeout) on each call based on how many calls have already been made?

I'm doing this sort of check when I make a call upon the API (throttles being an object with all calls and their limits):

var isThrottled = function(method) {
    if (throttles[method].count >= throttles[method].limit) {
        if (moment().diff(moment(throttles[method].last), 'minutes') > 1) {
            throttles[method].count = 1;
            throttles[method].last = new Date();
        } else {
            return true;
        }
    }
    return false;
}

in calls...

    if (isThrottled('getMarket')) {
        return callback({ message: 'call throttled' }, null);
    }
    var inv = session.getMarket(marketId);
    inv.execute(function(err, res) {
        throttles.getMarket.count++;
...

This is working but not terribly clean. I was thinking the API itself could handle throttling though so the consumer needn't worry about it.. and this is where a queue approach may work nicely. That is, have a job process for each API call and throttle it to ensure a consumer never attempts to make more calls than they are allowed. I'm playing around with the idea and I think it is fit for purpose (but mainly because I can't think of another way).

So, in summary, I was more interested in how you dealt with throttling and if you had plans to incorporate it in to your module.

If nothing is planned, please close the issue. Thanks, again.

[AlgoTrader]

I have no problems with throttling

setInterval(doMarketsRefresh, 1000);

doMarketsRefresh is being called once a second, 60 per minute. All the market refreshes are called by timer, timer guaranties the frequency is below throttling threshold

The other strategies like calling 60 calls in a second then wait for a minute I consider non-relevant