search

meraki-analytics / cassiopeia

An all-inclusive Python framework for the Riot Games League of Legends API. Cass focuses on making the data easy and fun to work with, while providing all the tools necessary to create a website or do data analysis.
MIT License
552 stars 135 forks source link

rate limit seems worked bad with mutilthread download data from different region data at same time #183

Closed wuzechen closed 6 years ago

wuzechen commented 6 years ago

Hi, I am developing a match log collector, and it works well when I download data from the single region once time. I am trying to download all 4 regions data (NA EUW KR JP) mutilThreadly at the same time. The code is written as below. collector thread will create a single thread work well collector.

    threadJP = collectorThread(1, "XXX", "JP", "7.22")
    threadKR = collectorThread(2, "Hide on Bush", "KR", "7.22")
    threadNA = collectorThread(3, "Søren Bjerg", "NA", "7.22")
    threadEU = collectorThread(4, "FNC Rekkles", "EUW", "7.22")

    threadJP.start()
    threadKR.start()
    threadNA.start()
    threadEU.start()

Then I found the rate limiter doesn't work properly. I print out the call log. Here is it.

crate thread JP
crate thread KR
crate thread NA
crate thread EUW
Making call: https://cdn.rawgit.com/meraki-analytics/cassiopeia/a6829400/cassiopeia/patches.json
Making call: https://cdn.rawgit.com/meraki-analytics/cassiopeia/a6829400/cassiopeia/patches.json
Making call: https://cdn.rawgit.com/meraki-analytics/cassiopeia/a6829400/cassiopeia/patches.json
Making call: https://cdn.rawgit.com/meraki-analytics/cassiopeia/a6829400/cassiopeia/patches.json
Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/29524444
Making call: https://euw1.api.riotgames.com/lol/summoner/v3/summoners/by-name/FNCRekkles
Making call: https://kr.api.riotgames.com/lol/summoner/v3/summoners/5346522
Making call: https://jp1.api.riotgames.com/lol/summoner/v3/summoners/6165138
Making call: https://jp1.api.riotgames.com/lol/match/v3/matchlists/by-account/200402334?beginIndex=0&endIndex=100&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=0&endIndex=100&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matchlists/by-account/200402334?beginIndex=100&endIndex=200&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=100&endIndex=200&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matchlists/by-account/200402334?beginIndex=200&endIndex=300&queue=420
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=0&endIndex=100&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=200&endIndex=300&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matchlists/by-account/200402334?beginIndex=300&endIndex=400&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=300&endIndex=400&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=400&endIndex=500&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matchlists/by-account/200402334?beginIndex=400&endIndex=500&queue=420
Making call: https://euw1.api.riotgames.com/lol/summoner/v3/summoners/20717177
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=100&endIndex=200&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=500&endIndex=600&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matchlists/by-account/200402334?beginIndex=500&endIndex=600&queue=420
Making call: https://ddragon.leagueoflegends.com/realms/jp.json
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=600&endIndex=700&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162152010
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=200&endIndex=300&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162152010
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=700&endIndex=800&queue=420
call JP 1
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162369239
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=300&endIndex=400&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=800&endIndex=900&queue=420
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=0&endIndex=100&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162369239
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=900&endIndex=1000&queue=420
call JP 2
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162118275
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=1000&endIndex=1100&queue=420
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=400&endIndex=500&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162118275
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=1100&endIndex=1200&queue=420
call JP 3
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=1200&endIndex=1300&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162705548
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=1300&endIndex=1400&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162705548
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=100&endIndex=200&queue=420
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=500&endIndex=600&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=1400&endIndex=1500&queue=420
call JP 4
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/161804893
Making call: https://kr.api.riotgames.com/lol/match/v3/matchlists/by-account/4261255?beginIndex=1500&endIndex=1600&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/161804893
Making call: https://ddragon.leagueoflegends.com/realms/kr.json
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3031171643
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=600&endIndex=700&queue=420
call JP 5
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162340772
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3031171643
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162340772
call KR 1
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3017883513
call JP 6
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/161969605
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=200&endIndex=300&queue=420
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=700&endIndex=800&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3017883513
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/161969605
call KR 2
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3012546393
call JP 7
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162171710
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3012546393
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=800&endIndex=900&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162171710
call KR 3
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3015377592
call JP 8
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/161254306
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3015377592
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/161254306
call KR 4
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=900&endIndex=1000&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3026669143
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=300&endIndex=400&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3026669143
call JP 9
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/161974248
call KR 5
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3055118526
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=1000&endIndex=1100&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/161974248
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3055118526
call KR 6
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=1100&endIndex=1200&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3030211588
call JP 10
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/163199728
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3030211588
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/163199728
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=400&endIndex=500&queue=420
call KR 7
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3050611395
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=1200&endIndex=1300&queue=420
call JP 11
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/163205355
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3050611395
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/163205355
call JP 12
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162893599
call KR 8
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=1300&endIndex=1400&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3052412984
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162893599
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3052412984
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=500&endIndex=600&queue=420
call JP 13
call KR 9
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162665728
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3025824471
Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43874214?beginIndex=1400&endIndex=1500&queue=420
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3025824471
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/162665728
call KR 10
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3033825642
call JP 14
Making call: https://ddragon.leagueoflegends.com/realms/na.json
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3033825642
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/163557440
Making call: https://na1.api.riotgames.com/lol/match/v3/matches/2646476413
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=600&endIndex=700&queue=420
Making call: https://jp1.api.riotgames.com/lol/match/v3/timelines/by-match/163557440
call KR 11
Making call: https://kr.api.riotgames.com/lol/match/v3/matches/3052673209
Making call: https://na1.api.riotgames.com/lol/match/v3/timelines/by-match/2646476413
Making call: https://kr.api.riotgames.com/lol/match/v3/timelines/by-match/3052673209
call JP 15
Making call: https://jp1.api.riotgames.com/lol/match/v3/matches/162180650
call NA 1
Making call: https://na1.api.riotgames.com/lol/match/v3/matches/2660974080
Making call: https://euw1.api.riotgames.com/lol/match/v3/matchlists/by-account/23989840?beginIndex=700&endIndex=800&queue=420

After these calls, collecting is stopped, or limited.

"Note that rate limits are enforced per region. For example, with the above rate limit, you could make 20 requests every 1 seconds to both NA and EUW endpoints simultaneously."

Riot said that rate limits are enforced per region, and my rate is 100/2min. It means I can download almost 50 matches(1 match costs 2 calls, match, and timelines) at the same time, but absolutely it doesn't work like I expected.

Is it a bug?

jjmaldonis commented 6 years ago

Hey we haven't done much testing for multithreaded apps, although I'm a bit surprised it didn't work. So yeah this is probably a bug. If you've got more code that I can use to reproduce this issue that'd be great (so that I can just copy-paste for testing exactly your problem).

It's the holiday so we might not get to it right away, but hopefully we'll have time to look into it in a week or so, and if the bug isn't too bad hopefully we can fix it quickly.

wuzechen commented 6 years ago

I'm sorry to disturb you during the holiday. I wrote the code base on the match_collection.py example, and do sth more about collecting KDA and some other features. And mutil thread part code is below

class collecterThread (threading.Thread):
    def __init__(self, threadID, initial_summoner_name, region, patch):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.initial_summoner_name = initial_summoner_name
        self.region = region
        self.patch = patch

    def run(self):
        print("crate thread {0}".format(self.region))
        while True:
            data_collecter.collect_matches(self.initial_summoner_name, self.region, self.patch)

if __name__ == "__main__":
    #cass.set_riot_api_key("")  # This overrides the value set in your configuration/settings.
    cass.apply_settings("setting.json")
    #cass.set_default_region("JP")
    # data_collecter.collect_matches("Hide on Bush", "KR")
    threadJP = collecterThread(1, "XXX", "JP", "7.22")
    threadKR = collecterThread(2, "Hide on Bush", "KR", "7.22")
    threadNA = collecterThread(3, "Søren Bjerg", "NA", "7.22")
    threadEU = collecterThread(4, "FNC Rekkles", "EUW", "7.22")

    threadJP.start()
    threadKR.start()
    threadNA.start()
    threadEU.start()
jjmaldonis commented 6 years ago

No worries, a couple of emails is hardly a bother :)

Thanks for the code too. Will check it out soon!

jjmaldonis commented 6 years ago

Hey sorry for the long delay. This took a bit to figure out. It should be fixed now.