[FEATURE] Support for additional mod-sites to keep track of versions?

[Kotlett]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe.

It's hard to keep track of the versions with non-modhub modifications. Especially with a big collection there are more then 75% of mods not listed on modhub.

Also versions are often not the newest on modhub (because of their approval system i guess).

Describe the solution you'd like

Is there anyway, you can add an support for different mod-sites?

For example, https://forbidden-mods.de/ has almost every mod, that exists for this game. If you could add an additional check like "if mod doesn't exist on modhub, check on http://...", it would be great.

Even if there's no api on this site, maybe the app could just search the html for the "versions" tab and extract the mod-version. Or keep a part of the version-related html in a cache to compare for ANY changes? Hope it's possible to understand, what I'm trying to suggest ^^

The user could manually enter a direct link for each mod, so the app knows, where to check for new versions.

Anything else?

For example - direct link to the "versions section" of an arbitrary mod:

"https://forbidden-mods.de/filebase/file/36556-fs22-steyr-multi-serie-2016/#versions"

[jtsage]

So, I needed to do some deep thinking on this. It probably isn't possible - at least not in any way that would be useful.

1.) The ModHub detection works with a site scraper, it's what it's sometimes a little behind. I wish there was a better way to get this data, but there really isn't. In a future version, I'm going to be also supporting FS < 2022 mods, on the up side, the modhub is closed for those now, so there is no need to ever update that data. But I can only do this method at all because the modhub records are predictable - the ID never changes, and it's tied to the name of the ZIP file - one of which I have, and the other I can compute.

2.) On the other hand, the URL from forbidden mods (or any of the other even less reputable sites) is not predictable from any of the data I already have - I'd assume the modder sets part of it when they initially upload stuff. In your example, the phrase Steyr Multi Serie 2016 doesn't exist at all, Steyr Multi Series does, but not the year, and certainly not the unique identifier that appears before it 36556

3.) The issue of scraping a user-entered link has 2 big issues, and a few minor ones.

• The internal technical one first - mod records in Mod Assistant are transient - if any part of the mod changes, the record is regenerated. This is to make sure that the display in MA always reflects what you really have on your disk. It would hurt the user experience a lot of the wrong version of a mod was shown, or it was missing the slight title change, etc, etc.
• The external technical problem is that scraping websites is slow and error prone. For this to work, I would have to build a table of Mod ZIP name -> website entries, and then scrape all of those entries. For that to take place, you are talking about potentially hundreds of web requests to load the main interface - or it could be deferred to the mod details page, which is a workable solution, but I'm not sure how useful it would be - you'd have to open every single mod to see if it had an update, which isn't much different from how most folks do it now. This also pre-supposes that the website in question doesn't have something set up to prevent a scraper from working (many do), and that the resulting webpage is going to be in a readable format - it's pretty easy for a human to scan a page, see a "Versions" title and recognize that the string "v1.0.0.1" is newer than the one they have - it's a lot harder for a scraper that has no concept of how the page is structured.
• The final, albeit less major, issue I see is that manually inputting the mod site for each mod is going to be very, very time consuming. In a perfect world, I could do some sort of online database component for this - but that only works until
  • 2 modders call their mods the same thing - FS22_JD_7R.zip is guarantied to be a unique file name if I pull from modhub, when you factor in all the folks who post simple edits and beyond the case is much different
  • Someone enters a link that isn't the modders preferred download site - there are a lot of trashy mod sites out there that re-host content without permission, FSG in general is pretty against it, and not something we'd ever want to encourage - and a internet based user-generated database would probably do exactly that.

For what it's worth, this was something I played with in the early days of development, although it was geared more towards getting versions from GitHub for some of the very popular mods hosted there (Universal Autoload, AutoDrive, CoursePlay to name a few) - it turned out to be significantly more processing and programming overhead than it was worth.

As to your point of offering a possible link or search on where to "check" - that's something I tried to do with the Folder Checker this is based on - it was a bit of a nightmare. Even just keeping it to opening a good search string in a search engine there was a fair bit of discussion over which search engines should be included, which should not, and how to "best" form the search query. I did not go down the road of including 3rd party hosting sites as there are far more sketchy sites than not, then there is the "why does this one get included but not mine" arguments. It's also hard to search most of those sites - I did a quick test, and with the example you posted, neither the file name nor the title as it appears in the modDesc would bring up that mod. Curiously, Steyr Mutli Serie did.

Anyway, sorry for the novel length reply, but I figured we may want to refer back to this in the future - I hope it explains my concerns adequately, and I'm more than happy to continue the discussion, either here, or if you prefer, a fair few of the FSG Modders hang out in our discord for far too long. (The link is handily at the bottom of the Mod Assistant window!)

[Kotlett]

Hey mate! Thanks for your really in-depth reply. I absolutely get your points.

As an (ex)webprogrammer myself, I know exactly how hard it is to "simply grab something from a page" and to make sure, everything works as intended, even with 19675753 possible failures on their site and so on. ;)

Just thought, maybe you have other knowledge/options/ideas for this. :)

Even without "proper version tracking", the mod assistant ist still incredibly helpful, cause i already can see each mod version with a preview picture in a simple list, instead of opening every single zip and moddesc.

In case a version check query on websites would ever be possible, i would integrate it just as an "manual on top option". Checking every mod from every page on startup isn't possible - that's a fact, no deny in this. But IF people could track all their versions, i bet they don't mind clicking an "check all Mods now!" Button and wait 30 minutes or so, of they have to do this only every few days. At least for me, clicking and just (long)waiting for the outdated list is more convenient, then search each mod individually in the web and compare the version.

But that's only my background thinking, as i completely get your point and of course it WOULD be a hell lot of work to get something reliable working.

In fact, that's the reason i suggested an "by the user entered link": there will be ALWAYS much error potential with these things, so a full automation would be even more difficult.

Further thinking; just a simple textbox on each mod, fillable by the user would do the trick and would be an improvement: i could enter my own link for each mod (where i downloaded it), and then open this page in-browser with a single click on a button to check the version myself! Even not 100% failsafe in case a file is down or they change the links, but that's on my own cap as a user then.

And nobody HAS TO manually fill something in - total up optional, just for people, who want to save their own link to a specific mod, and can then open them via a simple button right next to the displayed version..... Or something like that...

Would THAT be an option?

Because it would still be MUCH faster, then google every mod name, search for the right website, the latest version and so on. Just "click, look, close" after you entered the link for the first time.

Ps: already checked discord, but i find the channel a bit overwhelming and didnt know, where to discuss mod assistant there. ^^

[Kotlett]

Little addon:

For mods, which are not on modhub: an option to enter a manual link and open it via a button on the same place .... Instead of modhub. This should be intuitive!?

Maybe this option even for modhub mods (some sort of "optional overwrite"), because modhub often has not the newest version.

[jtsage]

Of the many options discussed here, I'd be most to lean towards providing a way to add your own link. It would end up tied to the mod "shortname" (that's the filename without the .ZIP extension).

I'm not sure if this will be doable in time for the next release, but I can see the utility for it, so will likely add it. Like previously mentioned, it'll be local to the end user only, and I can't see me having the time to implement any sort of scraping system - it's just to error prone to be workable I think.

[Kotlett]

That sounds great and will absolutely do the trick! Optional, user enterable and local textbox with a little "open it" button on the list frontend (like where the modhub button is) is all it needs.

To afterwards just click on each mods "open custom link" button and have a quick look on the opened page will be a lot quicker, then search for every mod manually in Google and navigate through, until you can finally compare the version numbers. :)

Ps: shortname should do it, as nobody should have same zip names with different content in the same folder of their hdd. Is this a unique identifier? So, if i tie "link xyz" to one mod in a collection, it will be tied to the same shortname in other collections too? That would be prefareble, i think. 🤔

[KITT3000]

Other sites like ModHub don't interest me much, but GitHub does, eg. Courseplay releases an update on MH with some larger versions and on GitHub it can be updated several times in one day, such a version checking function would be an interesting option.

[jtsage]

Other sites like ModHub don't interest me much, but GitHub does, eg. Courseplay releases an update on MH with some larger versions and on GitHub it can be updated several times in one day, such a version checking function would be an interesting option.

I agree with you there, from a personal standpoint. I played early on with keeping a list of those popular github repos and tracking updates (UAL, AD, CP, I think maybe the REA stuff gets updated there first?) - The only downside I could really see was I wasn't sure if I wanted the update flag set when the github version was newer - on my single player games I sure do, but on the servers we usually keep with the modhub version to keep it downloadable by the end user. I think probably this will get implemented in some fashion, but it may just be a set of options in the right click context menu - "Set source website" and "Open source website". Maybe for the really big github ones I could pre-populate on those, as it really is only a handful.

[KITT3000]

You can add something like this in options:

Check for updates:
------------------ Only official ModHub
------------------ ModHub and GitHub/GitLab
------------------ Everywhere

[Kotlett]

For me personally, the simple "textbox-with-a-button" addon is all you need. Especially because you can also just add a github page on that. ;) So it makes it universally for every mod and you have to develop MUCH less then any other solution, that only covers 2% of all mods. Everything additional is just bonus. :D

Just my 2 cents off course. ;)

[jtsage]

Here's what I've come up with, it's not bad.

Additional Flag for a site when it's set

Right click to access the set / open portion

The set modal dialog.

This will get pushed in a few days probably, I wanna spend some time testing it. Fwiw, I did prepopulate Universal Autoload, AutoDrive and Courseplay - those are the big ones on github that I know of.

[Kotlett]

All the REA mods: https://github.com/900hasse

(But I'm not aware, of he is still updating these)

I'm excited to see the update. Maybe you can add a direct-click button for the "open source" function but the rest seems really convenient. This would push usability even more for big collections. :)

[jtsage]

Included in 2.1.2, releasing right now.