Cached data not invalidated with latest updates

[gimpyestrada]

I just downloaded the new beta release, it says version 11.0 beta 1241

It looks like the change is getting detected backwards. You can see in the screen shot that it is pointing from newer version to older version.

[Willy-JL]

i doubt those are real updates. it looks more like the thread had the wrong version number, and mods have fixed it, thus resulting in an "update" that seems to be the wrong way around. the fact that this happened now at the same time for a game from 2022, one from 2023, and one from 2024, is further evidence of this.

[gimpyestrada]

That's not correct, Giant Guardians updates every 2 weeks. There absolutely WAS an update to 0.3.68, but the tool is confused.

[Willy-JL]

i see. its not backwards. its a logic issue with how the cache api functions, that unfortunately im not sure can be fixed.

• cache api indexed these threads on nov 16
• threads were updated on nov 17
• the uploader that updated the threads, published the update using the date that the content was released, not that it was added to f95zone
• this means, it does NOT show on latest updates page (not until you scroll dozens/hundreds of pages back)
• cache api uses latest updates to detect when to invalidate the cache for updated games
• thus, it did not see the update, and so it did not invalidate the cache, so you receive old data until cache expires (cache expiry is 7 days so nov 23)

as for looking like a downgrade for you, this is because you switched from a tool version that uses direct thread parsing, to the new version that uses my cache api. if this previous data was not present on your machine from the previous mode of refresh, you would not have seen this, instead the update wouldve been picked up 1 week later when the cache is expired.

that too, is not ideal, but im not sure anything can be done about it. the forum does not provide any way to check for updates in order of being published to f95zone, only by time that uploaders select. and it looks like some uploaders use the date of the content being released (to patreon or other) and not the date of it being uploaded for f95zone, which can be several weeks or even years in the past.

requesting the forum for just the version numbers again and again will undoubtedly hammer the forum's servers.

using a shorter time for cache invalidation is also not possible. since now, all requests are done by my server to f95zone instead of thousands of users hammering the forum, it means that my server as an individual has to do many more requests than the average user had to, so to compensate they need to have a longer cache expiry time.

[Willy-JL]

i asked Sam about it, looks like in this case it was an honest mistake by a new uploader that recently got promoted, who selected an update time in the past rather than when it was uploaded. should not happen again, or very rarely at most.

for now, ive cleared the cache for these 3 games so you wont have to wait a week for cache expiry.

i proposed some ideas to Sam that would help mitigate the issue, if it were to ever happen again, but seems like they are not possible to implement with the current state of their backend.

however he did have a different idea, he's allowed my cache api to make larger requests to their version api, so i will be able to run a sanity check every 12 hours to find version numbers we have that are different than ones on the forum and invalidate the cache for those. this way, if anything were to go wrong with monitoring latest updates (like in this case), the game information will be outdated for a maximum of 12 hours instead of 7 days.

[Willy-JL]

issue is now mitigated with #184 deployed to cache api, updates will be picked up at most 12 hours late, in worst case scenario. for most cases, updates will be picked up within 5 minutes via latest updates api.