How to scan the whole market without hitting limits

[dan-gamble]

Given the advised limits:

Generally speaking, you should send no more than 500 requests per hour and 5000 requests per day. Be somewhat human. If you encounter a captcha, try to answer/solve it as soon as possible.

And there can be up to 1,500,000 auctions on the market at one time what would be the best way to cover most of the market?

With the 500 / hour rate you get 16 * 500 = 8,000. This isn't close to 1,500,000 :D

I'm thinking more in the context of something like Futhead / Futbin where they have prices and BINs for every player on the market.

You could increase the number of accounts but that gets quite expensive quite quickly :)

[syndac]

Although you'd have to ask futbin and futhead to confirm, I believe general consensus is that they use a ton of different accounts. Easily affordable for huge sites like them that probably make a decent amount in ad revenue.

Without a ton of accounts, I don't believe there is a way to check the whole market. Even just limiting it to golds would be difficult. However, you don't need 1 request per auction. Just per search. You might be able to get decent info if your search were something like Gold/LaLiga/ST. I don't believe the API would return everything that matched that, but it would return enough results to help build some market data. And that would be done in just 1 request.

[dan-gamble]

That's what i figured.

When you say return enough results to build some market data, what do you mean? Wouldn't you still get a random 16 gold, ST, from LaLiga so it would be hard to get something like the lowest 5 BIN's for x player?

Thanks for your insight though, it is helpful :)

[syndac]

I mean that running that search, or let's even say you just ran a search with no filters....I'm not sure it would return every single auction that met that criteria. I don't know how many pages of results the fut API is good for. I'd test, but my account is temp banned :(.

You might be able to test how many results get returned. Just run a search with super broad filters and do print(len(items))

Getting the 5 lowest BINs for a player wouldn't be hard at all. Let's say their futbin price is 15,000. If you did a search for that player and set max bin of like 18000, you should only get a handful of results. Then just take the 5 lowest BINs from there.

If you wanted to do this for a TON of players, you'd just have to play with your search filters until your search is returning some kind of balance between not too many results and enough different players in the results to be useful.

[dan-gamble]

Gotcha!

I'm quite new to this API but using it wouldn't i just get 16 results everytime according to: https://github.com/futapi/fut/blob/master/fut/core.py#L818.

Or do you mean make this number quite high to get lots of results? If that's the case i imagine that would get picked up on quite quick.

Unless there another method in the API i haven't seen that would return something more along the lines of what you're talking about :)

[syndac]

I'm quite new to this API but using it wouldn't i just get 16 results everytime according to: https://github.com/futapi/fut/blob/master/fut/core.py#L818.

To be honest, I'm not sure if page_size == # of results or not. I just haven't tried running searches that way. I wouldn't change it...the readme mentions it's risky.

And I don't know of another method. I'd just run a broad search and print(len(items)). That'll tell you exactly how many results it returns. If you do, report back here. That's good info.

[dan-gamble]

Running:

self.f = fut.Core(os.environ.get('EA_EMAIL'), os.environ.get('EA_PASSWORD'), os.environ.get('EA_SECRET_ANSWER'), platform='xbox')

print(len(self.f.search('development', level='gold')))

Produces:

❯ ./manage.py fut                                                                                                                                                                                                          (env: futily) 
16

[syndac]

That's unfortunate. I guess it only returns the first page of results, then. I'm not sure exactly how the auction search results work in the web app....whether it returns all results or just returns that page's results and only loads the next page when you click for it. You'd have to ask @oczkers or inspect the server responses to figure that out.

However, digging through the fut api a little, it looks like it's set to 16 because that's what the consoles return. If you run the search in the web app, how many can you see on the first page? (again, I can't do this myself due to temp ban). If you see more than 16, you should be fine setting page_size to whatever that number is.

Otherwise, you'd have to do something like:

pages=5
n=0
while n <= pages:
    self.f.search('player', level='gold', start=n)
    sleep(randint(5,10))
    n+= 1

That should search the first 5 pages with 5-10 seconds between each, but it's still 1 request per search. You'd really have to play with your filters to find the right balance between requests and results.

[dan-gamble]

Yea, the web app returns 16 results a pop from their API.

I guess it really is a case of just having lots and lots of accounts or some kind of leniency to be able to do scraping on a large scope.

Currently i have it looping over the pages like you showed there but with only ~500 requests an hour i don't get much accomplished 😄

I guess i'll have to try find another way to do something like this on a large scale :)

[oczkers]

I assume sites like futbin can have arrangement with bot service sellers - some players have 2-3 days old info but some are refreshing every few minutes so maybe when somebody is hunting for X player this info is getting pushed to futbin. Just an idea but it seams cheapest.

You can increase result length above 16 (50 was max in last year) but keep in mind that's "anomaly" and can help ea track illegal activity ;-)

[dan-gamble]

Thanks Piotr. That makes a lot of sense and isn't a bad idea :)