search

nrb / destiny-info-bot

A set on of functions to scrape destiny sites and report them to IRC.
GNU General Public License v3.0
1 stars 1 forks source link

Cache weekly/daily info #5

Open nrb opened 9 years ago

nrb commented 9 years ago

The weekly nightfall and heroic information could be cached for a period of 6 days. This would reduce the dependence on external calls and blocking the bot.

Similarly, the daily and bounty info could be cached for a shorter period of time.

This all should probably go in a new module that wraps the scraper calls, and is consumed by the destiny_bot functions.

nrb commented 9 years ago

For the vendors, we will need to cache based on the vendor name, not just the 'bounty' function name.

nrb commented 9 years ago

Another thing to consider - caching these values for too long will cause stale values to exist in the cache. So, the weekly information could be cached for a maximum of 6 days, but only that cache isn't set right before the applicable reset.

For example, caching the nightfall for a day or more on a Monday will mean the cache value will not match after the weekly reset.

A more reasonable thing might be to just cache for some number of hours, like 6 or so.

Since this is going to be in a self-contained module with caching specifically for the bot, it could try to use some date math smarts to fudge dates/times on the border, but I'm not sure that's really messy.

nrb commented 9 years ago

One final thing that might be nice to have - appending a message when a response is first cached vs when it's pulled from cache.

onelson commented 9 years ago

If we switch to pulling data from bungie.net (#11) then this should probably be simplified as a single daily cache mechanic (all daily events and weekly events would be pulled down and parsed from a single payload).

nrb commented 9 years ago

True, though I still worry about when it would happen. I'm leaning towards using some kind of scheduled mechanism within helga (the bot this is based on) so that this could happen each day, rather than a per-request basis.

onelson commented 9 years ago

Yep. A daily scheduled "continuous cache" makes sense.

Owen Nelson

On Tue, Apr 28, 2015 at 10:52 AM, Nolan Brubaker notifications@github.com wrote:

True, though I still worry about when it would happen. I'm leaning towards using some kind of scheduled mechanism within helga (the bot this is based on) so that this could happen each day, rather than a per-request basis.

— Reply to this email directly or view it on GitHub https://github.com/nrb/destiny-info-bot/issues/5#issuecomment-97152377.

nrb commented 9 years ago

I think that we can actually use the response from Bungie's API to tell us this, which allows us to see what bungie's reset dates/times are.

Using these fields ('nightfallResetDate', 'heroicStrikeResetDate', 'dailyChapterResetDate', 'dailyCrucibleResetDate'), we should be able to make more efficient caching.

Since we'd be fetching information daily that contains the weekly information as well, it's likely easiest to update everything on a daily basis, possibly at the first request after the expiry. Also, since we have a consolidated request for most of this info now, only the API response would need to be cached, further simplifying the code base.