leveler.py - Switch to using a cache of pre-calculated required_exp level values

AznStevy / Maybe-Useful-Cogs

Random cogs for a discord bot.

MIT License

29 stars 31 forks source link

leveler.py - Switch to using a cache of pre-calculated required_exp level values #60

Closed travisgroth closed 2 years ago

travisgroth commented 6 years ago

This PR attempts to improve run time of a few looped calls to _required_exp() on larger servers.

Defines a list to cache calculations for each level
Creates a method that returns the sum totals that had been calculated inline previously; caching results for a given level if it needs to calculate a higher level
Pre-calculate up to level 100 at startup
Replaces any looped calls to _required_exp() with a single call to _total_required_exp()

AznStevy commented 6 years ago

Yeah, I'm aware. It's not just big servers, it's for big bots since it had to go through all users. I'm probably just going to create a server based scheme so the database only needs to check for the server I'd instead of 100k users lol. I don't think the time is only because of that loop, though you're also right that it might be better to cache the values. I do think the pr that I ignored does it as a single sum so it should be fairly fast.

travisgroth commented 6 years ago

I couldn't fully isolate it without a lot of test data, but I mimicked the logic in a test script and found that the loops doing the _required_exp() call per-level + per-user accounted for almost all of the run time for 'top/top10'.

The implementation in this PR reduced my test script run time from ~4.5 minutes to 1 second on a minimal set of 25k 'user' data structures. That's all in memory and ideal case (no mongo or json parsing), but I'm fairly confident its representative of a serious real world improvement. Caching is absolutely the naive approach here but it appears fairly effective.

AznStevy commented 6 years ago

Oh ok, that's exciting. I'll try it myself when I get back in a bit.