Large MyAnimeList lists cause RateLimitExceededError

[purplepinapples]

I modified the code from here:

import time
import aniffinity

time.sleep(2)

success = False

for i in range(5):
    print(i)
    try:
        af = aniffinity.Aniffinity("purplepinapples", base_service="MyAnimeList")
        affinity, shared = af.calculate_affinity("lanblade", service="MyAnimeList")
        print(affinity, shared)
    # Rate limit exceeded. Halt your script and try again
    except aniffinity.exceptions.RateLimitExceededError:
        time.sleep(10)
        continue

    # Any other aniffinity exception.
    # Affinity can't be calculated for some reason.
    # ``AniffinityException`` is the base exception class for
    # all aniffinity exceptions
    except aniffinity.exceptions.AniffinityException:
        break

    # Exceptions not covered by aniffinity. Not sure what
    # you could do here. Feel free to handle however you like
    except Exception as e:
        print("Exception: `{}`".format(e))
        break

    # Success!
    else:
        success = True
        break

The code above fails with a RateLimitExceededError.

Even increasing the time.sleep to over a minute does nothing, since in order to process purplepinapples or lanblade (which I chose because of their length), it takes around 50 load.json requests, meaning its bound to exceed the API limit sometime in this loop.

I have a crude fix on my fork, for which the code above succeeds.

Perhaps accepting an argument in aniffinity.Affinity and aniffinity.calculate_affinity like wait_time that waits between paginated requests?

[erkghlerngm44]

Hiya

Yeah, the API being used here is a massive PITA when it comes to rate limiting. I've encountered a similar problem to yours when writing the code for that section, and I wasn't really sure how to handle it.

Good idea about the argument in aniffinity.Aniffinity and aniffinity.calculate_affinity - I'll see if I can get that implemented.

Not sure how far it would help though - the last time I tried using the API, it had problems retrieving my list (containing around 350 entries) with a wait of 60 seconds between requests. Hopefully they get around to putting up a decent official API one day...

[purplepinapples]

FYI there are plans for the API, but no date yet.

Thanks for the fix. The code above now works since it defaults to 2 seconds, but I can't pass wait_time to calculate_affinity. Passing wait_time to aniffinity.Aniffinity works fine.

[erkghlerngm44]

Ooh, interesting. How long ago was that written?

As for passing wait_time to aniffinity.Aniffinity.calculate_affinity, there's no need. When you pass it to aniffinity.Aniffinity, that time is used for all paginated requests for that instance. calculate_affinity calls comparison, which calls the endpoint and passes the wait time you specified when creating an instance of the class.

[purplepinapples]

How long ago was that written?

It was on April 5th.

Thanks for the clarification, that makes sense.

[erkghlerngm44]

No problem!

It was on April 5th.

Thanks, hopefully the API won't take too long to be released.

[sam5558]

Hello, I'm currently writing a little script that automates friends name collection with affinity and shared anime. My script starts collecting these info, but as you said in your comment, the script fails after a random time... It is indeed a RateLimitExceededErrror. I've played with the value of wait_time but it doesn't seem to fix anything. Here's where it fails

  File "./anifinity.py", line 32, in <module>
    myaffi[i],myshared[i] = af.calculate_affinity(myfriends[i], service="MyAnimeList")
  File "/home/zen/.local/lib/python3.6/site-packages/aniffinity/aniffinity.py", line 228, in calculate_affinity
    pearson = calcs.pearson(scores1, scores2)
  File "/home/zen/.local/lib/python3.6/site-packages/aniffinity/calcs.py", line 34, in pearson
    return float(num / den)
decimal.InvalidOperation: [<class 'decimal.DivisionUndefined'>]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./anifinity.py", line 38, in <module>
    except aniffinity.exceptions.RateLimitExceededError:
NameError: name 'aniffinity' is not defined

[purplepinapples]

You probably imported it as from aniffinity import Aniffinity, right?

If you want to use the exceptions you'd have to do something like:

import aniffinity
try:
    aniffinity.Aniffinity("Guts__", base_service="MyAnimeList", wait_time=3)
except aniffinity.exceptions.RateLimitExceededError:
    print("Exceeded Rate Limit!")

or, instead:

from aniffinity import Aniffinity
from aniffinity.exceptions import RateLimitExceededError
try:
    Aniffinity("Guts__", base_service="MyAnimeList", wait_time=3)
except RateLimitExceededError:
    print("Exceeded Rate Limit!")

[sam5558]

Thank you for your quick reply @purplepinapples , i'm trying your second proposition; this will take some time (20min+) to get each username's affinity and shared value (85 friends :D ). Anyway i'll tell you if it works !

EDIT : It worked ! Thanks again ;)

[erkghlerngm44]

Hi @sam5558 ,

Apologies for the late response - this isn't my main GitHub account so it may take a while for me to see anything that happens here. Glad to see that you've managed to get it working (thanks @purplepinapples !). When I get a chance I'll update the documentation to make this clearer.

I'm curious about something you wrote in the jikan repo issue. I don't want to bother the people there, hence the reason I'm asking this here:

I've also noticed that the Pearson's correlation coefficient doesn't give the exact percentage (sometimes i get 2 or 3% of difference between what myanimelist states and what the script outputs)

Would it be possible for you to give me an example of this happening? It may just be a floating-point issue but I'd like to be sure. Thanks!

[sam5558]

False alert ! I've retried today and values seems to concord. I was either really tired or this issue vanished :D. Thank you anyway. I've managed to get all my friends and display affinities in a table (ordered by affinity), so if you're interested in my code you can visit this link @erkghlerngm44 @purplepinapples ... I'm a complete newbie in development so don't expect something awesome.