Avoid querying IFWiki whenever a game page is loaded

[salty-horse]

When a browser loads the game page, it makes an AJAX call to IFDB, which makes an HTTP call to IFWiki, looking for a wiki page for the game in question (based on the first IFDB).

This is done every time a page is loaded, which is unnecessary. As discussed in iftechfoundation/ifdb#123, the result should be cached for a reasonable period of time, and the IFWiki link should be returned as part of the page.

• Should the caching be done in a new column in the database?
• What is a reasonable period of time for which to save positive/negative results?
• Which action should cause the cache to be cleared?

[alice-blue]

• What is a reasonable period of time for which to save positive/negative results?

I can think of only a few reasons why a previously-existing game page on ifwiki might become unavailable:

• The title of the ifwiki page has changed. (Sometimes there is more than one game with the same title, and the pages could get renamed "Game Title (by Author)" to differentiate them. In this case I would think there would still be a disambiguation page for "Game Title," but I suppose it's possible someone could forget to make a disambiguation or redirect page.)
• ifwiki is down
• vandalism

Since none of these things happens very often, maybe a positive result gets saved for 6 months or so.

Not sure about how long to save a negative result.

[dfabulich]

I pulled some data on this today.

• We don't query IF Wiki on every page load… we only do it when the game has a known IFID (because that's what we use to query IF Wiki).
• There are 6,236 games in IFDB with an IFID
• Of those, 1,322 games have IF Wiki pages that have the same IFID. (That's 21%.)
• We're not generally in the habit of creating new IF Wiki pages for individual games, at least not in the last few years. The most recent game with an IF Wiki page is Tavern Crawler, in 2020. (Somebody created IF Wiki pages for all the IFComp 2020 games, but no one has done that for any of the subsequent IFComps.)

Therefore, I think it makes sense to have a very long cache time, for positive and negative results.

It might even make sense to do a one-time import, adding IF Wiki links as External links to the 1,322 games that have them, and then stop automatically querying IF Wiki at all. Users could manually add/remove links to IF Wiki if/when new game pages were added to the wiki.

Alternately, if IF Wiki could conveniently dump out a list of its IFIDs (?) we could probably query that once a day / once a week, and use that cache instead of directly pinging IF Wiki all the time.

[salty-horse]

Here's an API call to get all IFID prefixes in IFWiki: https://www.ifwiki.org/api.php?action=query&format=json&pslimit=500&list=prefixsearch&pssearch=IFID

It's in pages of 500, so needs extra calls. Mediawiki documentation.

[alice-blue]

IFWiki is in the process of putting together a new games database now. I'm not sure how that might affect how IFIDs are organized/stored/etc. on IFWiki.

[salty-horse]

Here's an attempt to query IFWiki directly from the browser instead of going through IFDB, so there's no need for the ifwiki-check API.

It's giving an error about a missing Access-Control-Allow-Origin - Is that missing on IFWiki? Is there away to "allow" it with headers on IFDB?

[dfabulich]

1024 moved the request to IFWiki, but I think we should still not bother pinging IFWiki all the time.

My favorite answer is to have a scheduled job that updates an IFDB cache table of IFWiki links.