Automatically detect MO directories

[qsniyg]

This would allow for minimal editing (possibly none) on the user's part. The idea would be to automatically detect all of the MO game directories from the MO installation directory (which would have a few predefined paths to look for, and be user configurable).

It would also detect which game is to be VFS'd based on the CWD and the ModOrganizer.inis.

[jarrard]

Maybe it would be better to scan ModOrganizer.ini and determine paths from that. Only issue is if you use mapped drives then script won't know that D:\ means /mnt/Gamesdrive/ (unless you can define that somewhere)

[qsniyg]

Maybe it would be better to scan ModOrganizer.ini and determine paths from that

Sorry, yes, that's what I meant :)

Only issue is if you use mapped drives then script won't know that D:\ means /mnt/Gamesdrive/

I'm not quite sure what you mean by mapped drives, sorry... isn't this something that can be fixed by using ~/.wine/dosdevices though, or am I misunderstanding?

[jarrard]

Mapped drives is something you can specify in wine configuration window. You can define what D/E/F etc drives map to. The benefit of doing it that way is to keep ini and configuration files consistent between linux and windows so there isn't any confusion.

For example when I booted back up into windows10(or a VM) to test some things, all my mo2 paths were wrong because there is not mnt/drive folder under windows, its D: or E:

[qsniyg]

Ah I see, so they should be mapped under dosdevices then :)

[ajventer]

I don't agree with this. The RIGHT way to do it, which has been on my TODO list for some time - is to remove the Paths dictionary from teh code and put it in a config file which can be specified with a commandline flag. That way users can actually have multiple and use the script for say FO4 and SkyrimSE alike - and not have to keep two copies of the script for the job.

[qsniyg]

Good idea. What if we were to merge the ideas then?

As in, automatically generate the config file (maybe with user input, maybe via a command-line option, or maybe quit after the first run, instructing them to edit the file as they please and re-run) using the ModOrganizer.ini? That way, users don't have to insert paths manually to use it, but if they want/need to, they can.

About the configuration files, should it be one configuration file, or a configuration file per game directory? If it's the latter, we could provide an option to automatically generate all of them (if they don't exist), and then whenever you run movfs4l.py somewhere within the game directory, it'll detect the configuration file at the top level of the game directory and use that.

[ajventer]

All good options - some much harder than others. I don't think autodetection is really viable, there is just too many ways a user can set up their system. Where would Plugins.txt go ? The path is somewhat fixed on Windows (though tied to user accounts which are not), but that can be altered through wine's mount option - it could live ANYWHERE on the filesystem. That's why I let users set that up manually. We can't just read it out of MO - MO is operating within wine, we are not.

I would suggest just shipping the sample config as is - but in a seperate file, possibly using YAML rather than JSON to make it easier to read. A more interesting question is the other one you raise. One config with sub-modules or a config per game ? Either way we need to support a parameter to select which game to VFS/UNVFS at runtime so I don't think the user-experience changes on that level. It then becomes a case of what is easier for a user to manage. My suspicion now you mention it is that a single config may be the simpler option. Something like this:

SkyrimLE: MOROOT: '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/ModOrganizer/SkyrimLE', mods: '{MOROOT}/mods/', overwrite: '{MOROOT}/overwrite/', profiles: '{MOROOT}/profiles/', skyrim: '{PREFIX}/drive_c/Steam/steamapps/common/Skyrim Special Edition', plugins.txt: '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/Skyrim/plugins.txt' FO4: MOROOT: '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/ModOrganizer/Fallout4', mods: '{MOROOT}/mods/', overwrite: '{MOROOT}/overwrite/', profiles: '{MOROOT}/profiles/', skyrim: '{PREFIX}/drive_c/Steam/steamapps/common/Fallout 4', plugins.txt: '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/Fallout4/plugins.txt'

And then the script can be run something like: ./movfs4l.py --game SkyrimLE UNVFS

[jarrard]

I'm not 100% sure but I think Mod Organiser 1 still works for SkyrimLE UNVFS, its just the new mod organiser that has a issue. (could be wrong)

[ajventer]

It does but MO1 is not being maintained and hasn't been able to sign into nexus since they switched to https last year.

I can't actually sign in with MO2 since they switched to the cookie based auth last month - wine doesn't play nice with that it seems but at least that will potentially be fixed

[jarrard]

Yeah its a real shame we can't just go back to MO1 for UNVFS and keep the new stuff MO2 has.. I dunno what the whole reason for the change was for, except to comply with new stuff windows 10 does, but mo1 unvfs worked just as fast.. Just a puzzling change....

[ajventer]

The change was actually for a different reason altogether. With FO4 and SSE for the first time Bethesda started building 64-bit game engines.

MO1 is a 32-bit application, and MO uses library injection (not too different from LD_PRELOAD tricks on linux) to fake the VFS. You can't do that if the library and the application are different architectures. Injecting 32-bit library routines into a 64-bit application would cause it to crash -badly.

So when these games came out, MO couldn't support them. It hadn't had a maintainer in some time (the guy who wrote it is now the lead dev on Vortex). So some other devs decided to fork it, make a 64-bit version and keep it up to date. They also basically rewrote some entire parts of the app -notably the UVFS library to get much better performance that can only be done with 64-bit systems. This works because windows DOES allow injecting 64-bit library calls into 32-bit apps (it just can't be done the other way around).

The problem was, in doing so, they used a feature of windows 10 that wine doesn't support. And hardly anything else uses it. There is an open bug for the issue, and somebody posted a patch in there with a "Please test". I haven't actually tried it, mostly because these days I run the games via proton but running MO via proton is not currently a setup I have any idea how to do, so I use MO in it's own bottle with it's own wine and let my script take over injecting the mods into the game dirs for use with Proton.

[jarrard]

Makes sense. I hope some day soon MO2 64bit can work under wine, so close...

[qsniyg]

I don't think autodetection is really viable

Sure, not everything can be autodetected, but it could fill in those that it guesses, and leave the user to fill in the rest.

but that can be altered through wine's mount option - it could live ANYWHERE on the filesystem

Yes, but we can follow links, no? AFAIK (and correct me if I'm wrong), the directories under wine don't change, only whether or not it's symlinked.

It might not work for 100% of cases, but I do believe auto-generation would be quite helpful for the 90% of the time it does work. It's not the ultimate solution, but it would be a nice helper, to at least fill in some of the values. It'd definitely lower the barrier for usage for most people.

Where it gets complicated is, as you say, with user accounts, especially if someone used both a wine, and a proton prefix (which creates users/[username] and users/steamuser). However, although it is a possibility, it isn't the norm either. In those cases though, I would suggest it would just avoid generating that portion (and possibly spit out an error).

That's why I let users set that up manually

Absolutely, by no means am I stating auto-generation should replace manually setting it up, I just believe it would be a nice way to easily set up some (or all) values, and let the user modify them later if needed.

Basically, automatic generation would just replace writing a prebuilt YAML file by a dynamically generated one at first launch (and possibly via a command-line option as well).

possibly using YAML

I agree, it also supports comments which would be quite important in making it easier to understand how to use.

My suspicion now you mention it is that a single config may be the simpler option.

Yeah I think so too.

Either way we need to support a parameter to select which game to VFS/UNVFS at runtime so I don't think the user-experience changes on that level.

Absolutely. However, again on this front, how about automatically detecting the game based on the current directory, if the parameter is missing? Something like:

>$ python movfs4l.py UNVFS
No game specified, detected Skyrim (use --game GAMENAME if you want to specify another game)
...

There is an open bug for the issue, and somebody posted a patch in there with a "Please test".

Are you referring to this bug? While I haven't tried it yet, I don't think it will fix the issue. Check the source code here, it seems it will just print something to the log and move on if the function does not exist.

[qsniyg]

Actually, how about TOML instead of YAML? It's like INI, but with comments (in fact, we could just use .ini as the extension).

I personally find it much easier to edit things with TOML, and plus indentation is not required (indentation is only easy to edit with a good text editor, something not everyone might have). INI is also the format used by ModOrganizer, and the Besthesda games as well, so people will more likely be familiar with it.

Using the format you suggested:

[SkyrimLE]
MOROOT = '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/ModOrganizer/SkyrimLE'
mods = '{MOROOT}/mods/'
overwrite = '{MOROOT}/overwrite/'
profiles = '{MOROOT}/profiles/'
skyrim = '{PREFIX}/drive_c/Steam/steamapps/common/Skyrim Special Edition'
plugins_txt = '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/Skyrim/plugins.txt'

[FO4]
MOROOT = '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/ModOrganizer/Fallout4'
mods = '{MOROOT}/mods/'
overwrite = '{MOROOT}/overwrite/'
profiles = '{MOROOT}/profiles/'
skyrim = '{PREFIX}/drive_c/Steam/steamapps/common/Skyrim Special Edition'
plugins_txt = '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/Skyrim/plugins.txt'

Another idea could be to cascade settings:

[default]
MOROOT = '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/ModOrganizer/{GAME}'
mods = '{MOROOT}/mods/'
overwrite = '{MOROOT}/overwrite/'
profiles = '{MOROOT}/profiles/'

[SkyrimLE]
GAME = 'SkyrimLE'
skyrim = '{PREFIX}/drive_c/Steam/steamapps/common/Skyrim Special Edition'
plugins_txt = '{PREFIX}/drive_c/users/{USERNAME}/Local Settings/Application Data/Skyrim/plugins.txt'

Just an idea though :)

[ajspandigital]

That's why I let users set that up manually

Absolutely, by no means am I stating auto-generation should replace manually setting it up, I just believe it would be a nice way to easily set up some (or all) values, and let the user modify them later if needed.

A good middle ground may be a config-generator option which tries to autodetect a game, asks the user to specify the paths, and offer defaults based on what it could autodetect. Shouldn't be too hard to make it ignore anything it already has in the config.

Absolutely. However, again on this front, how about automatically detecting the game based on the current directory, if the parameter is missing? Something like:

>$ python movfs4l.py UNVFS
No game specified, detected Skyrim (use --game GAMENAME if you want to specify another game)

That's only viable if the user keeps separate copies of the script in the game directories, which would mean you can't just default to the current dir for config files the way we do for the activity log file - which would also need to be expanded to have separate ones for each configured game. Either you need to now specify the path to a central config file, or you need to have individual config files anyway. Seems to defeat the purpose of the config option entirely. Not to mention that the user would have to update those scripts individually to get the latest version. I would like a situation where they can just clone the repo, and use it from the clone point straight up. And just do a git pull to update the script itself when we release new features.

There is an open bug for the issue, and somebody posted a patch in there with a "Please test".

Are you referring to this bug? While I haven't tried it yet, I don't think it will fix the issue. Check the source code here, it seems it will just print something to the log and move on if the function does not exist.

Mmm, those test result shots at the end were added since I last looked. I never actually looked at the patch itself. Yes, just showing an error message would probably not resolve the problems there - wine would have to actually support this API at some point.

[ajspandigital]

Actually, how about TOML instead of YAML? It's like INI, but with comments (in fact, we could just use .ini as the extension).

I personally find it much easier to edit things with TOML, and plus indentation is not required (indentation is only easy to edit with a good text editor, something not everyone might have). INI is also the format used by ModOrganizer, and the Besthesda games as well, so people will more likely be familiar with it.

I have no problem with any of those ideas. My one concern though with either yaml or toml is that they require third-party python modules. I have thus far successfully avoided ever requiring one. Even ujson is only used if the user already has it - and the fallback to the stdlib json works just fine otherwise. It just makes it easier on the user if this simple script doesn't have any dependencies they have to acquire third-party.

[qsniyg]

A good middle ground may be a config-generator option which tries to autodetect a game, asks the user to specify the paths, and offer defaults based on what it could autodetect.

I agree, although I think many of the paths can be autodetected. For example, if the user is running it with a WINEPREFIX set (or inside a wineprefix), it can check for C:\Modding\MO2. If it's found, it could then show something like:

MO2 Installation detected at: ~/.wine/drive_c/Modding/MO2
Do you wish to specify a custom path instead? (y/n)

If it doesn't find one, then it could just prompt the user for the directory.

Either you need to now specify the path to a central config file, or you need to have individual config files anyway.

Good point, and yeah I agree, having it all in the git repo makes it much easier.

But as you mentioned, the _log.json also needs to be considered. Since it's different for each game, there has to either be one massive _log.json that's updated (an idea I really don't like, as any corruption would corrupt all games, not to mention the performance and memory cost), or multiple _log.jsons. Those could either be in the game directory, or in the repository. Both have their pros and cons.

Personally I'd argue to place them in the game directory, as if ever you need to remove and re-clone, it would be awful to accidentally lose those files.

My one concern though with either yaml or toml is that they require third-party python modules

Good point, I hadn't thought of that. How about https://docs.python.org/3/library/configparser.html then (INI)? Turns out you can use comments there, just not inline, so it should be fine.

[ajspandigital]

Yes we would definitely need to go for per-game activity logs. The resources required to parse those logs (which get REAL big for big modding setups) if you have multiple games managed would be insane. Sure a modern PC CAN do it - but why slow the user down for no reason.

I haven't made up my mind between gamedir vs scriptdir on log location though. As you say - pros and cons both ways, have to think about it.

Ini may be the best solution yes, it's simple - well known and easy to read.