search

encratite / LibOfLegends

This is a C# networking library that pretends to be the League of Legends Air client. It was developed for the purpose of stats tracking. It depends on a modified version of the FluorineFX library available in the FluorineFXMods repository. It also depends on NilSharp, a random general purpose library. Released under the terms of the GPLv3.
GNU General Public License v3.0
84 stars 22 forks source link

Best way to populate a server database? #21

Closed Batatech closed 11 years ago

Batatech commented 11 years ago

Simple question - what is the best way to populate a server database (like lolking, elobuff) with all the summoners in a server? When I query the server for a summoner it takes about 300ms for the entire operation to complete.

encratite commented 11 years ago

There are three primary methods, I suppose.

  1. poll their end of game stats HTTP server by bruteforcing game IDs and then obtaining stats for games through that, this is teh only method to possibly cover an entire server but also prohibitively expensive and easily IP bannable
  2. using players from ranked ladders as seeds for queries... actually, I just realised, they reduced the ladder on the site to only the top 100 players? That's terrible, so you'd have to use lolsummoner or whatever for that
  3. using players from manually entered queries as seeds

Seed meaning a starting point for spiderig oepratinos in which you gradually expand into other summoners in their match histories. In general you want to perform as many queries per second as you can, like 10-20., per connection to the LoL server. They might have implemented throttling/disconnects/IP bans for that by now, I have no idea, I haven't messed with this in months really. RiotControl was configured to perform queries at a low rate, only one every 2-5 seconds IIRC

Batatech commented 11 years ago

Hey, thank you for the quick reply!

The seeding part wasn't the problem - I had figured it out, same method as you point out. Unsurprisingly enough one single player from the Challenger ladder can generate a gigantic tree (over 100k so far). I imagine one single active player is enough to populate almost the entire database, except for small clusters of inactive players?

My problem is that each query takes around 300ms to complete, so I'm getting about 3 queries per second. Should I try to perform multiple queries simultaneously then?

encratite commented 11 years ago

Oh, yes, of course, you need 50-100 simultaneous connections, likely from multiple IPs.

encratite commented 11 years ago

But you can also perform multiple queries in parallel on the same connection! You don't need to wait for one to finish!

Batatech commented 11 years ago

Alright, I'm gonna try that. Thank you very much. I'm really not experienced with that right now (first time actually dealing with RPCs and databases ever, I'm just curious about some statistics which are not present in any site. Most sites only deal with the NA, EUW and EUNE servers) so I wasn't really sure how much I should be pushing the server before risking a throttle/ban.