Fallback to thread versions if dedicated API returns N/A

[colombeen]

I've added games as well as mods, but for the mods I never get a version number and they only seem to update when I trigger a full recheck.

examples:

• https://f95zone.to/threads/109357
• https://f95zone.to/threads/106166

[FaceCrap]

As you may know, somewhere in it's lifetime F95Checker switched from retrieving the threads during a Fast Refresh to a dedicated API made by the site admin. This was done to reduce the hammering on the site, which occurred from GET'ing anywhere from a few dozen to hundreds or even thousands of page threads. Now, the thing is, and I may be mistaken but I'm fairly sure I'm not. A quick test I just did running the sourcecode through a debug process reveals that those two mod threads do not show up in the returned list from the dedicated API calls during a fast refresh. If that is always so I don't know, I only tried a few mods besides those two, and none showed up. However, when I added a comic thread id in the API call, it did return the 'version'.

Which means that the only time the actual thread page itself will be retrieved is, as you noticed, during a full check, so that the first post can be parsed for information.

@Willy-JL will be able to explain what the deal with the version is. Since those two and a few other mods I tried, didn't turn up in the data from the API call, I expected it might gotten extracted from the thread during a full recheck, but I suspect the inconsistent format is why it's not picked up then.

That it even is able to detect if a mod is updated is because there is at least one item on each thread page which is consistent with every post, and that is the Last edited date bottom-right. If the checker can't make head or tails from the Updated date listed in the post, it'll revert to that that field. For the two mods you mentioned, it is that date which is picked up. It will however pick up the Updated date if it uses the correct format, as can be seen for this mod https://f95zone.to/threads/143177/

[loongemperor]

I think there is more to this than just the API Even if I manually force a full rechecks it does not work. And even on a full recheck every single mod fails to parse the version. They all say N/A for their version. As seen here

Most mods use the Syntax of

Mod Version: 1.0
Game Version: 1.0

Although some of them just say

Version: 1.0

the same way a normal game does. others (like universal renpy mod) https://f95zone.to/threads/universal-renpy-mod-urm-2-0-3-mod-any-renpy-game-yourself.48025/ omit the Game Version line. and just say

Mod Version: 2.0.3

I think that:

A. the full recheck should have two fallbacks for version. if Version: is not found, fallback to Mod Version: and if that isn't found fallback to Game Version:

B. The above does not explain why standalone mods who only present Version: don't parse the version either. so there seem to be a bug there.

C. if a thread is not found via the API. perform a full recheck for that thread only. probably with its own separate timeout that checks less often than the API refreshes.

D. if the API is specifically excluding all threads tagged with MOD. then a mod updater module should check those threads separately using full refresh

[Willy-JL]

A. the full recheck should have two fallbacks for version. if Version: is not found, fallback to Mod Version: and if that isn't found fallback to Game Version:

it does not parse the thread for version numbers at all. the point of the new api sam (site admin) made is that we dont need to fetch the threads, since that creates a lot of overhead on their servers. instead, he made a dedicated api using a much faster redis instance rather than going through the main forum, that only serves version info. before, it was a clusterfuck with all the threads formatting slightly differently, because even if to you it looks like just "Version: 1.2.3" there is spacing, formatting, bold, and all the HTML retardedness preventing correct parsing. since this api is now a highly reliable source for a correct and stable version field, it relies on this to decide when to run full rechecks. but as you saw, some threads dont have this info, or they change other details when the version doesnt (say downloads, image, tags...), so it does full rechecks every once in a while anyway.

making it parse versions from the thread too in case it doesnt come from the dedicated api is an option, but i will not make it fetch always for things that dont get served by the dedicated api. making it do a full refresh more often for things not served by the dedicated api might be an option, depending on how much of the code would need to be reworked.

[Willy-JL]

that should cover most of them

[loongemperor]

it does not parse the thread for version numbers at all. the point of the new api sam (site admin) made is that we dont need to fetch the threads, since that creates a lot of overhead on their servers. instead, he made a dedicated api using a much faster redis instance rather than going through the main forum, that only serves version info.

Just to clarify, I was not talking about the API there. I was talking about what happens when you right click and select "full recheck" Are you saying that the "full recheck" also uses the API?

[FaceCrap]

Are you saying that the "full recheck" also uses the API?

It does, but where before it just set the version to N/A for threads which didn't get returned by the API call, the parser now also looks for the version in the thread itself. And if it can't be found in the thread itself (or it isn't formatted correctly) it falls back to the version in the thread title (provided it is formatted correctly here too). And this doesn't apply to just mods, some games also didn't return a version in the API data, these now also show a version instead of just N/A.