spike: investigate symlinks

[pikdum]

• loadorder.txt and plugins.txt for bethesda games
  • like the post-deploy scripts
• .ini for bethesda games, so vortex doesn't complain
  • like the fallout new vegas pre script
• how to migrate from post-deploy scripts to this

I think there are some non-Bethesda games that do similar, too. Maybe:

• XCOM2
• GTA V
• Sims 4

[SirStig]

Thankfully I do have all those games, just have to download, mod them all, and see what vortex does.

As far as simplifying, I wasn't quite sure how to do that, I'm more proficient in C#, HTML, CSS, and Javascript, never messed with unix before, but seems quite simple.

Migrating, probably would have to add rm commands to post-install.sh that removes everything that is not needed anymore.

If your going separate script, not running off of the Vortex.desktop, can just create a Symlink to the desktop for this. I know vortex comes from your other repository, my thought was just adding the sh "/home/deck/.pikdum/steam-deck-master/vortex/update-games.sh" to it fairly simply. Maybe thats not how it works, didn't really look into it.

Edit:

All I did to the vortex.desktop file was add sh "/home/deck/.pikdum/steam-deck-master/vortex/update-games.sh" & before launching vortex.

[pikdum]

Wonder if the logic should be added to https://github.com/pikdum/vortex-linux instead, then. That way it can use Javascript, and could be added to the vortex-linux CLI.

Tricky part is that we can't assume the Steam Deck's game paths - maybe there's a way to get those from Steam somehow?

[SirStig]

Wonder if the logic should be added to https://github.com/pikdum/vortex-linux instead, then. That way it can use Javascript, and could be added to the vortex-linux CLI.

Tricky part is that we can't assume the Steam Deck's game paths - maybe there's a way to get those from Steam somehow?

Game Paths? Do you mean the Compatdata paths? The ID's for all games are the games links. So https://store.steampowered.com/agecheck/app/489830/ the 489830 at the end is the games ID and also what the folder for that game would be called.

Edit:

Also Oblivion seems to also just use Loadorder.txt, and plugins.txt, I'll check other games, but would be safe to assume that its probably the same for all games. As far as .ini files, it seems all the games have those in the Vortex directory. Skyrim, Oblivion, and Fallout.

[pikdum]

Maybe thats not how it works, didn't really look into it.

Basically vortex-linux is a CLI with some basic commands for things like:

• downloading Proton
• setting a Proton version
• installing Vortex
• setting up that Vortex .desktop entry
• launching Vortex

Wrote it in Javascript since it was easier to maintain + reason about than a huge bash script. vortex-linux is made to work on any Linux, with nothing Steam Deck specific in it. Can check the README.md for more details.

This repo mostly just wraps around it and configure things specifically for Steam Deck.

Game Paths?

The actual install paths, since they could really be anywhere depending on how somebody set up their drives. Like I could have my steam games on /mnt/ssd/steam/ or similar. With Steam Deck, we know they're most likely either on internal storage or SD card unless somebody did something really weird, so got away with hardcoding the paths here.

but would be safe to assume that its probably the same for all games.

Yeah, pretty sure all those Bethesda games do it the same. Don't remember Morrowind, though.

[SirStig]

Maybe thats not how it works, didn't really look into it.

Basically vortex-linux is a CLI with some basic commands for things like:

* downloading Proton

* setting a Proton version

* installing Vortex

* setting up that Vortex .desktop entry

* launching Vortex

Wrote it in Javascript since it was easier to maintain + reason about than a huge bash script. vortex-linux is made to work on any Linux, with nothing Steam Deck specific in it. Can check the README.md for more details.

This repo mostly just wraps around it and configure things specifically for Steam Deck.

Game Paths?

The actual install paths, since they could really be anywhere depending on how somebody set up their drives. Like I could have my steam games on /mnt/ssd/steam/ or similar. With Steam Deck, we know they're most likely either on internal storage or SD card unless somebody did something really weird, so got away with hardcoding the paths here.

but would be safe to assume that its probably the same for all games.

Yeah, pretty sure all those Bethesda games do it the same. Don't remember Morrowind, though.

Well actually even in UNIX there is a way to just search for a directory. find /dir/path/look/up -name "dir-name-here" is an example. Can use this to just search the steam deck, and external drive and find the steamapps folder, or just a specific game.

As far as .ini's go, it seems that there's alot of stuff in the Documents folder, under My Games. Not necessarily needed though from what I can see even without us creating Symlinks or copy pasting the files, Skyrim at least has the exact files. I mean in long term may be good to copy/paste or create symlinks for it just in case there is any need.

Find Steamapps directory in Internal drive: find $HOME -type d -name steamapps

Find Steamapps directory in external drive: find $/run/media/mmcblk0p1/ -type d -name steamapps

[pikdum]

Some notes:

• ~/.steam/steam/steamapps/libraryfolders.vdf has all the libraries, and which games are in which
• the format is a bit unique, but can probably parse with https://www.npmjs.com/package/vdf-parser in vortex-linux

So, could do something like:

$ ./vortex-linux setupSymlinks

This would call a setupSymlinks function, that:

• we have a mapping of game ids to what symlinks it needs
• parses libraryfolders.vdf to get installed games + their locations
• loops through and sets up symlinks

As an aside, I feel like STL tried to do something similar, but not sure it really worked.

[SirStig]

Example STEAMAPPS=(find / -type d -name steamapps)

Search entire computer, all drives for steamapps folder, and output the directory link to the var STEAMAPPS.

[SirStig]

Some notes:

* `~/.steam/steam/steamapps/libraryfolders.vdf` has all the libraries, and which games are in which

* the format is a bit unique, but can probably parse with https://www.npmjs.com/package/vdf-parser in vortex-linux

So, could do something like:

$ ./vortex-linux setupSymlinks

This would call a setupSymlinks function, that:

* we have a mapping of game ids to what symlinks it needs

* parses libraryfolders.vdf to get installed games + their locations

* loops through and sets up symlinks

As an aside, I feel like STL tried to do something similar, but not sure it really worked.

Interesting, still have the issue though of finding the steam library instead of assuming the location. Issue with really anyway is that with linux, we have to use wine/proton. The issue there is finding any folder name steam, steam library, anything there are tons of them because of different bottles.

Easiest thing to do would honestly be to just open a window for the user to select the steam folder themselves.

Edit: Another issue to deal with would be multiple steam folders. If libraryfolders.vdf has locations to games that fixes the issue, if not then if you have a steam library folder on multiple drives for different games your running into an issue.

[pikdum]

Interesting, still have the issue though of finding the steam library

That libraryfolders.vdf has it all, just need to parse it.

The issue there is finding any folder name steam, steam library, anything there are tons of them because of different bottles.

There's really only two things:

• compatdata, basically the wine prefix or fake windows install per game
  • should always be in ~/.steam/steam/steamapps/compatdata/
  • uses game ids as directory names
  • should be no issue consistently getting this
• actual game folder
  • location can be looked up through that libraryfolders.vdf
  • can have multiple libraries, but all the info will still be in that file
  • but doesn't really use anything consistent as game directory name, so we'd need to keep a mapping
  • game directory name is the installdir in the ~/.steam/steam/steamapps/appmanifest_${gameid}.acf VDF file

~~STL maintains a mapping of game ids here: https://github.com/sonic2kk/steamtinkerlaunch/blob/925bc318582831b03fa880bcebffe3c375cf2309/misc/vortexgames.txt Not sure if they have anything that'd match a game id to what it's folder is called, though. Like for Monster Hunter Rise, the game ID is 1446780 and the game folder is MonsterHunterRise.~~

[SirStig]

Every single game has a steam_appid.txt which only contains the App ID inside.

[pikdum]

There's also the appmanifests:

Like ~/.steam/steam/steamapps/appmanifest_489830.acf

They're also VDF files, and have the installdir name we need.

$ cat ~/.steam/steam/steamapps/appmanifest_489830.acf | grep installdir
    "installdir"        "Skyrim Special Edition"

So I think we should be able to get everything needed to do https://github.com/pikdum/steam-deck/issues/27#issuecomment-1590630314.

[SirStig]

This is an interesting find Vortex

[pikdum]

Yeah, I think technically Vortex can be built and ran natively on Linux. But I don't think there has been any effort to make these parts run natively:

• 3rd party tools, like FNIS and Outfit Studio
• the actual game launching

So easier to just run the whole thing in Wine/Proton.

[SirStig]

I’m pointing more towards the Wine-installer branch wine-instller

Edit: This does everything you were wanting to do.

[pikdum]

Ah, yeah. Seems similar to what vortex-linux does for most of it. Seems like some extra pieces it does:

• parses libraryfolders.vdf and makes game symlinks in Vortex's compatdata/wine prefix
• sets up registry entries so games are auto-discovered by Vortex

Looks like it'd be hard to maintain those bash scripts, though.

[SirStig]

Ah, yeah. Seems similar to what vortex-linux does for most of it.

Seems like some extra pieces it does:

• parses libraryfolders.vdf and makes game symlinks in Vortex's compatdata/wine prefix

• sets up registry entries so games are auto-discovered by Vortex

Looks like it'd be hard to maintain those bash scripts, though.

From what I read through from the symlinks it is almost exactly what you're wanting. I don’t really see any reason for it to be needing. To be maintained though. It’s pretty simple

[pikdum]

I think those symlinks are just for setting up the game paths, along with keys.reg, so Vortex finds them automatically. Not seeing anything symlinking .ini files, or loadorder.txt and plugins.txt, etc. unless I'm looking in the wrong place.

[SirStig]

I think those symlinks are just for setting up the game paths, along with keys.reg, so Vortex finds them automatically.

Not seeing anything symlinking .ini files, or loadorder.txt and plugins.txt, etc. unless I'm looking in the wrong place.

Your right, what it actually does is link the My Games folders, and roaming folders and so on.

PREFIX_PATH" ]; then link_sub_targets \ "$VORTEX_PREFIX/drive_c/users/$USER/My Documents/My Games" \ "$CURRENT_PREFIX_PATH/drive_c/users/steamuser/My Documents/My Games"; link_sub_targets \ "$VORTEX_PREFIX/drive_c/users/$USER/AppData/Roaming" \ "$CURRENT_PREFIX_PATH/drive_c/users/steamuser/AppData/Roaming"; link_sub_targets \ "$VORTEX_PREFIX/drive_c/users/$USER/AppData/Local" \ "$CURRENT_PREFIX_PATH/drive_c/users/steamuser/AppData/Local"; link_sub_targets \ "$VORTEX_PREFIX/drive_c/users/$USER/AppData/LocalLow" \ "$CURRENT_PREFIX_PATH/drive_c/users/steamuser/AppData/LocalLow";

[pikdum]

I'd have to dig deeper in the code to see what the exact links its making are. Might be similar to what I was trying here: https://github.com/pikdum/vortex-linux-bash/blob/20e4be2811771ad8f4929463ac305e6ef9f0432c/vortex-linux#L308 My implementation was a bit buggy, though.

[SirStig]

Okay, messed around a ton but what do you think of this https://github.com/SirStig/Steam-deck-tests

Tested it on Skyrim LE cause I accidentally downloaded that instead of Skyrim SE but it works perfectly!

Might combine the two different .sh scripts. Other then that the only thing I found is becuase of the links Vortex suggests a path to the C: drive symlink for steam. But it refuses to do any deployment method unless you have the Vortex staging folder in J:/Vortex Mods which could just be caused by you mounting J:/

[SirStig]

So main differences:

• Post-install.sh Checks also if the vortex.exe exists instead of just .desktop in case somehow the vortex prefix gets deleted or something.
• Added a version to in theory eventually be used. Didn't want to do much work on what I wanted with that.
• Post-install.sh Checks for any old deploy.desktops and deletes them.
• install.sh Pulls version being installed from git, as well as completely deletes the Steam-deck-master folder before installing again to remove any old files.
• Added a loaderlibrary.json with a specific format for loaders. Here all you have to do from now on is add the Steam Game ID, Mod loader launcher .exe file name and games defualt Launcher name for it to automatically find the game, and change the mod loader launcher to be the same name as the games defualt launcher.
• readme.md Just fixed some grammer, and changed the parts that explain deployment and so on to match my changes.
• Update-vortex-library.sh Finds and locates the steam library, goes through all the currently installed games and creates symlinks for Documents, Local, LocalLow, Roaming, and the steam library folder.
• Vortex-Post-Updater.sh might merge this and Update-vortex-library.sh to the same file, but this one goes through the steam library locates all the games and checks the loaderlibrary.json and checks if any of the game id's match if they do it swaps the launchers.

Edit: Forgot I was going to have the loaderlibrary.json updated from git, so no updating or anything needed when the Vortex-Post-Updater.sh is launched. I'll quickly do that now.

Edit: Edit: Added maybe a dumb way to update loaderlibrary.json but done.

[pikdum]

But it refuses to do any deployment method unless you have the Vortex staging folder in J:/Vortex Mods which could just be caused by you mounting J:/

Interesting. So it's fine with the game on C:/ with the staging folder in J:/?

Update-vortex-library.sh Finds and locates the steam library, goes through all the currently installed games and creates symlinks for Documents, Local, LocalLow, Roaming, and the steam library folder.

Maybe setup-games.sh would be a better name for this? Seems like we could run it once immediately after installing Vortex, but we'd want to have it as a desktop shortcut so it can be manually run later too.

Vortex-Post-Updater.sh might merge this and Update-vortex-library.sh to the same file, but this one goes through the steam library locates all the games and checks the loaderlibrary.json and checks if any of the game id's match if they do it swaps the launchers.

Wonder if we could split this into two, maybe something like:

• use-script-extenders.sh
• use-original-launchers.sh - to revert

That way we could add both to the desktop, and people could easily swap.

Added a version to in theory eventually be used.

Not sure we really need a version at this point.

[SirStig]

Yeah, would be easy to have two separate launcher.sh that revert or change.

Names can be changed easily.

Think I might changed the loaderlibrary format to be the same as steams vdf.

Yes, not sure why but for whatever reason games are automatically found and fine under c:/ but the staging folder has to be under J:/

Edit: My reason for the version.txt was so when you do Update Pikdums tools it checks the version first and doesn’t just delete and download unnecessarily

[SirStig]

Having a version also gives people an idea of what version of Pikdums tools they are using. Could also be used to notify people of a new update.

[pikdum]

loaderlibrary.json

Yeah this could be a different extension, since it's not json right now. Not really sure what it is.

Yes, not sure why but for whatever reason games are automatically found and fine under c:/ but the staging folder has to be under J:/

What if you remove J:? It's a symlink.

Edit: My reason for the version.txt was so when you do Update Pikdums tools it checks the version first and doesn’t just delete and download unnecessarily

I'd rather keep it as-is.

Having a version also gives people an idea of what version of Pikdums tools they are using.

I'd want to use semver with https://github.com/googleapis/release-please, similar to what I'm doing in vortex-linux, then. Would need some reworking, though.

[SirStig]

Yeah, It was a txt file but I decided to use json just because txt felt weird.

I’ll change it to a .vdf with same format as steam.

All this was more so to get it all working.Don’t need a version up to you.

Didn’t try removing the J:/ symlink but was going to

[pikdum]

It's hard for me to follow bash scripts that I haven't written, but update-vortex-library.sh is making symlinks for the actual game directories somewhere in Vortex's program files, right?

Is that enough for them to be auto-detected by Vortex? Sort of expected to need registry entries too.

[SirStig]

It's hard for me to follow bash scripts that I haven't written, but update-vortex-library.sh is making symlinks for the actual game directories somewhere in Vortex's program files, right?

Is that enough for them to be auto-detected? Sort of expected to need registry entries too.

Yea, so what Update-vortex-library.sh does is, it locates the Steam library folder. Parses the library.vdf, pulls game prefixes, names, and locations from it. Then for every single one creates a symlink between LocalLow, Local, Roaming, Documents, and also the Steam library itself. Which actually works perfectly for vortex. Just have to launch vortex and click manage game and it’ll automatically just start setting it up. Doesn’t ask for location, and vortex says the games location is actually in the C:/ drive.

[pikdum]

How does it work with multiple games? Say I have Oblivion and Skyrim, which both need their loadorder.txt and plugins.txt to be symlinked. And probably any .ini files too?

[SirStig]

How does it work with multiple games?

Say I have Oblivion and Skyrim, which both need their loadorder.txt and plugins.txt to be symlinked.

And probably any .ini files too?

Hmm, I didn’t look into that, I would have to change the symlinks then. I messed around a bit and didn’t think of that. Would be an easy fix just have Vortex prefix folders symlinked to every single game prefix instead. I had it the other way around.

[SirStig]

Okay, let's look at it now.

Some things are a little messy. However I have changed the format of the loaderlibrary to be a vdf. Format is alot cleaner and easier to read. I cut a few corners in that department but it all works. Removed the version references, and version.txt. Added a script to swap back to vanilla launchers, however I wanted users to have full control over this. So I created arrarys and added the install paths, names, launcher, loader, and game id that are currently installed based off of the loaderlibrary.vdf.

It then uses those arrays, pulls a list of the currently installed games that we currently know to have script extenders, and ask the user to input the game id which is listed next to the games name to be able to select which game to revert back to vanilla.

I have not had time to try removing the symlink J:/ as all my times been spent coding

[SirStig]

Update:

Removing J:/ and Z:/ do not fix the staging issues. I believe it may have something to do with the symlinks of the directories. Maybe switching from symlink to hardlinking the steam library may fix it?

Update 2: Tried adding chmod o+x to vortex.desktop in the applications folder. Could need to do that to vortex-linux instead? Point was more checking if permissions issues are the issue

[pikdum]

Maybe switching from symlink to hardlinking the steam library may fix it?

Maybe, but hardlinking sounds like it could be a bad idea. For example, I don't think uninstalling games would work as expected then.

Point was more checking if permissions issues are the issue

I think it's more that Vortex wants them on the same drive, because otherwise hardlink deployment won't work. I'd guess it's probably a bit tripped up by the symlinks. Using J: and K: for the staging folders seems fine, though.

[pikdum]

Oh, did you have the game you were testing with installed on the SD card or internal storage?

Edit: Nevermind, probably internal storage because J:.

[SirStig]

Oh, did you have the game you were testing with installed on the SD card or internal storage?

Edit: Nevermind, probably internal storage because J:.

Yup Internal. Seems it's because Vortex can tell they are symlink directories. So it's fine for a game but for deployment methods to work such as a Hardlink, it has to be on the same drive. So it refuses to deploy unless its on the J:/ because it doesn't notice C:/ and J:/ are technically the same drive because of wine

[pikdum]

Want to make a pull request when you're happy with where it's at?

Might have some more feedback on the code, and I'll want to test it pretty extensively before merging.

[SirStig]

Yeah, links for updates, and so on will need to be changed. I have a few minor changes to make and I'll make a pull request