sonic2kk / steamtinkerlaunch

Linux wrapper tool for use with the Steam client for custom launch options and 3rd party programs
GNU General Public License v3.0
2.1k stars 70 forks source link

Adding Non-Steam Games Does Not Use The Right `userdata` Path with Multiple Accounts #1140

Closed WIFIDarthMaul closed 3 weeks ago

WIFIDarthMaul commented 1 month ago

System Information

Issue Description

Like the title states, I am having trouble adding Non-Steam Games to Steam via commandline (Konsole). I have been using Steamtinkerlaunch for quite some time now and was using it to add games via .sh and it worked great for the longest time but now no longer does. Any help?

sonic2kk commented 1 month ago

Are you using the Steam Client Beta out of interest? It did make some changes but it shouldn't have affected anything with shortcuts.

Also, did this happen after updating SteamTinkerLaunch and if so, do you know what the last working commit was? It worked for me about a week ago, after the initial Steam Client Beta update.

You will also need to provide a log of an attempted execution where you add a Non-Steam Game (you removed this section of the issue template). This may give some insight into what is going wrong.

sonic2kk commented 1 month ago

Just tested, addnonsteamgame still works over here. A log will be needed to investigate the problem. Any information you can add about when it stopped working might be helpful, too, to understand if this was may have been caused by a Steam update, a SteamTinkerLaunch update, or something else.

I also just tested by removing my shortcuts.vdf file entirely to see if perhaps the format had changed but was not impacting existing shortcuts.vdf files for some reason. This is not the case, a fresh shortcuts.vdf works perfectly fine. I will try one treated by Steam and will try writing into it, by removing the existing file, adding a Non-Steam Game via Steam, and then attempting to write into that file. I expect this to work, as on startup, the Steam Client does sanity checks on this file and some other files too.

WIFIDarthMaul commented 1 month ago

Hmm. I'm not sure if I am on the Beta. I will check. I'm on Steam Deck (Arch) btw. I was using 12.0 during that time and it worked but now doesn't. I tried the latest release and it doesn't work for me too. It must me some Steam update.

On Wed, Jul 10, 2024, 6:10 PM Eamonn Rea @.***> wrote:

Just tested, addnonsteamgame still works over here. A log will be needed to investigate the problem.

— Reply to this email directly, view it on GitHub https://github.com/sonic2kk/steamtinkerlaunch/issues/1140#issuecomment-2221674596, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJBOJ3PWTKFVQCRI3A4M2HTZLW5NNAVCNFSM6AAAAABKV2AGFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRGY3TINJZGY . You are receiving this because you authored the thread.Message ID: @.***>

sonic2kk commented 1 month ago

12.0 is very, very old. It's best to always use the latest commit to master, especially on SteamOS.

I'm on Steam Deck (Arch) btw

The Steam Deck is very different from a development target perspective than vanilla Arch Linux, please do not conflate the two and be explicit when opening issues about the operating system you are using.

SteamOS is no longer tested against and I will be removing advertisement of official support come v14.0.

Please test this on the Linux Desktop.

I tried the latest release and it doesn't work for me too.

If you are not using the latest commit to the master branch, you should try that.

It must me some Steam update.

The Steam Client Beta (on Linux Desktop) and Steam Deck Beta (specific to SteamOS), while usually close, are in fact different in various ways.


if I am understanding correctly, this stopped working even back on v12.0. It is probably a good idea to check master, as major changes have been made to Steam Shortcuts in that time.

sonic2kk commented 1 month ago

I just tested with a fresh shortcuts.vdf with no shortcuts, created by Steam. I then added a game from Steam, and used SteamTinkerLaunch to add a game. When restarting the Steam Client, both games (the one added from Steam and the one added by SteamTinkerLaunch) were still present in my library.

This may be an issue specific to v12.0, which was released in December of 2022, or with v12.12, which was released in March of 2023, both being extremely outdated and given there have been numerous changes to Non-Steam Games since then, it's possible that something was fixed in master that Steam now breaks on with those older versions. #902, maybe #942, and #951 come to mind, but in particular, #965 could be something here. I was surprised Steam didn't bork the file because these were missing to begin with. It is possible that now it is doing that.

I cannot reproduce this issue so far, if you have any other specific cases you'd like me to try, let me know. A log file from an attempted add of a Non-Steam Game when running from master will help too.

WIFIDarthMaul commented 1 month ago

I tried the master and still no luck. Here's the log generated: steamtinkerlaunch.log

sonic2kk commented 1 month ago

I don't think that's a log from right after running the Add Non-Steam Game command, it looks like it's from the Steam Client startup where it validates compatibility tools:

Wed Jul 10 07:13:24 PM CDT 2024 INFO - main - Checking command line: incoming arguments 'run /home/deck/.local/share/Steam/ubuntu12_32/../bin/d3ddriverquery64.exe -d3d12'
Wed Jul 10 07:13:24 PM CDT 2024 INFO - main - Checking command line: first argument 'run'
WIFIDarthMaul commented 1 month ago

I don't think that's a log from right after running the Add Non-Steam Game command, it looks like it's from the Steam Client startup where it validates compatibility tools:

Wed Jul 10 07:13:24 PM CDT 2024 INFO - main - Checking command line: incoming arguments 'run /home/deck/.local/share/Steam/ubuntu12_32/../bin/d3ddriverquery64.exe -d3d12'
Wed Jul 10 07:13:24 PM CDT 2024 INFO - main - Checking command line: first argument 'run'

Whoops. I updated the log link

sonic2kk commented 1 month ago

Thanks. I'm not seeing anything immediately jumping out in the log. The start command looks correct and it looks as though the function succeeds.

This is a shot in the dark: Are you using multiple Steam accounts, or did you add a new Steam account to your Steam Deck recently? If so, it's possible SteamTinkerLaunch is writing to the wrong shortcuts file.

If you do not have multiple accounts... Then I am not sure :-)


As a courtesy note (there could be several valid intentional reasons for this, just want to mention) that you are using --usesteamgriddb, new in master, but are not passing a SteamGridDB API key. You can find out how to set this up here. I was curious if this could potentially cause an issue (such as trying to write an invalid icon into shortcuts.vdf) but it worked even without a SteamGridDB API key and with an invalid one.

WIFIDarthMaul commented 1 month ago

You know what I did sign into a different Steam account about a week ago! I'll check

And no worries, I understand what the command does. And I did notice that I needed the API key. Funnily enough I was just writing code for an app that I'm developing that also requires an API key, different service though, so I'm familiar.

On Wed, Jul 10, 2024, 7:56 PM Eamonn Rea @.***> wrote:

Thanks. I'm not seeing anything immediately jumping out in the log. The start command looks correct and it looks as though the function succeeds.

This is a shot in the dark: Are you using multiple Steam accounts, or did you add a new Steam account to your Steam Deck recently? If so, it's possible SteamTinkerLaunch is writing to the wrong shortcuts file.

  • You can get your Steam UserID by opening the file at ~/.local/share/Steam/config/loginusers.vdf. Here there will be a list of users logged into the Steam Client on this account.
  • Look for the block with your AccountName. The number above the opening curly brace for that block should be quite long. This is your Long UserID. You can convert it to a short ID by running the following command from Konsole, replacing 11111111 with your Account ID. echo $(( 11111111 & 0xFFFFFFFF )). I recommend copying and pasting it to make sure the result is accurate. Running commands from a random person on the Internet is scary and I don't believe in doing that, so I'll try to explain what this does. Feel free to verify on your own as well. You don't need to provide any of the actual IDs here to me, this is purely for if you want to verify this for yourself.
    • echo basically just tells Bash, your shell, to print something out into the console window.
    • The $(( )) part is how Bash, the shell, denotes arithmetic operations. It tells Bash "interpret everything here as arithmetic and then give it back to me." So whatever is computed inside of $(( )) is returned as a number. For example, echo $(( 1 + 1 )) would print out 2. -The 11111111 & 0xFFFFFFFF bit is bitwise arithmetic to convert the number given into a different format. Bitwise arithmetic is something I don't fully understand, but it has been reverse engineered by people much smarter than me to know this is how you convert down to the, iirc, 32bit unsigned integer that Steam uses. We use this arithmetic for other things too like calculating the Non-Steam Game AppID.
  • Once you have your short User ID, let's illustrate that it's 123456, you can that User ID and go to this folder, ~/.local/share/Steam/userdata/123456/config, replacing 123456 with your shortened User ID that you just got back. This folder contains the shortcuts.vdf file and some other files (such as Non-Steam Game grids and localconfig.vdf that stores some other general per-user Steam Client configuration).
  • Check the date that the shortcuts.vdf was modified. If it was not very recently (i.e. when you last ran the command), then SteamTinkerLaunch is writing to the wrong location. From the log, I can see that Steam is writing to /home/deck/.local/share/Steam/userdata/1520722864/config/shortcuts.vdf. If this is not the right path then something is probably up here.
    • You could do some more tests, such as closing the Steam Client and re-running the SteamTinkerLaunch command, and then checking if SteamTinkerLaunch actually wrote into the file (did the number of bytes change, did the modified date change, etc - This can be seen from the "Details" view in the Dolphin file manager, the third button to the right after the forward/backward buttons, or Ctrl+3 on a physical keyboard).

If you do not have multiple accounts... Then I am not sure :-)

As a courtesy note (there could be several valid intentional reasons for this, just want to mention) that you are using --usesteamgriddb, new in master, but are not passing a SteamGridDB API key. You can find out how to set this up here https://github.com/sonic2kk/steamtinkerlaunch/wiki/SteamGridDB#api-key.

— Reply to this email directly, view it on GitHub https://github.com/sonic2kk/steamtinkerlaunch/issues/1140#issuecomment-2221785677, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJBOJ3P6FAKQZPJ2K7HRSBDZLXJ5TAVCNFSM6AAAAABKV2AGFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRG44DKNRXG4 . You are receiving this because you authored the thread.Message ID: @.***>

sonic2kk commented 1 month ago

This is something that has been in the back of my mind for SteamTinkerLaunch for a while (I've probably mentioned it buried in other issues by now) but we do need a way to specify the User ID. The default when no ID is passed should be to use the last logged in user. Right now SteamTinkerLaunch doesn't look at loginusers.vdf at all, it just guesses the directory by I think taking the one with the lowest value that is not 0 or ac but I am not sure. Instead we should take the User ID of the user in the block with "MostRecent" "1". This was not done before probably because parsing VDF files with Bash is a pain but has been implemented since then to read and write to config.vdf and localconfig.vdf specifically for Steam Shortcut-related functionality.

For specifying custom IDs - such as if you're logged into another account but have the Steam option enabled to prompt you for which account youu want to use, you might want to write into a specific user's shortcuts.vdf without having to open Steam and log in first, then restart Steam - adding it on the commandline is easy depending on the route taken; I'm unsure whether it should be direct ID or username. I am very much leaning towards ID but getting the short ID is not easy, so then it would have to be the long ID, but it's not even straightforward to get that without looking at loginusers.vdf yourself. The trouble with going with username is that they aren't guaranteed to be unique between accounts on the same computer. It is a bit tricky and I have other Non-Steam Game things I am looking to finalise, but it is something I have already planned to take a look at.

And once it's added to the commandline it can be exposed as an option on the UI fairly easily (using the username and User ID in a combobox entry in case we can't read loginusers.vdf properly for whatever reason).

If this issue is related to the last logged in user ID being guessed wrong, then unfortunately there is not a workaround currently, but this weekend I could try looking into changing the default behaviour to use the most recently logged in user which is one step in the right direction.

WIFIDarthMaul commented 1 month ago

You mad lad!

Looking at my main account's .vdf, I can see that it hasn't modified in some time, and logging into my alt account, welp... now there's a whole bunch of repeated Non-Steam Game shortcuts. So that was the issue.

On Wed, Jul 10, 2024, 8:33 PM Eamonn Rea @.***> wrote:

This is something that has been in the back of my mind for SteamTinkerLaunch for a while (I've probably mentioned it buried in other issues by now) but we do need a way to specify the User ID. The default when no ID is passed should be to use the last logged in user. Right now SteamTinkerLaunch doesn't look at loginusers.vdf at all, it just guesses the directory by I think taking the one with the lowest value that is not 0 or ac but I am not sure. Instead we should take the User ID of the user in the block with "MostRecent" "1". This was not done before probably because parsing VDF files with Bash is a pain but has been implemented since then to read and write to config.vdf and localconfig.vdf specifically for Steam Shortcut-related functionality.

For specifying custom IDs - such as if you're logged into another account but have the Steam option enabled to prompt you for which account youu want to use, you might want to write into a specific user's shortcuts.vdf without having to open Steam and log in first, then restart Steam - adding it on the commandline is easy depending on the route taken; I'm unsure whether it should be direct ID or username. I am very much leaning towards ID but getting the short ID is not easy, so then it would have to be the long ID, but it's not even straightforward to get that without looking at loginusers.vdf yourself. The trouble with going with username is that they aren't guaranteed to be unique between accounts on the same computer. It is a bit tricky and I have other Non-Steam Game things I am looking to finalise, but it is something I have already planned to take a look at.

And once it's added to the commandline it can be exposed as an option on the UI fairly easily (using the username and User ID in a combobox entry in case we can't read loginusers.vdf properly for whatever reason).

If this issue is related to the last logged in user ID being guessed wrong, then unfortunately there is not a workaround currently, but this weekend I could try looking into changing the default behaviour to use the most recently logged in user which is one step in the right direction.

— Reply to this email directly, view it on GitHub https://github.com/sonic2kk/steamtinkerlaunch/issues/1140#issuecomment-2221819876, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJBOJ3MY5UH4ABBLCHW37UDZLXOGLAVCNFSM6AAAAABKV2AGFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRHAYTSOBXGY . You are receiving this because you authored the thread.Message ID: @.***>

sonic2kk commented 1 month ago

Heh, I call it intuition.

The full fix for this is two parts:

  1. Fix how we get the Steam userdata folder, because right now it is not actually basing it on anything other than directory order. I think a reasonable default is the last logged in account. This won't cover all cases, but I think it is a reasonable default, at least better than what we have now. To my understanding there is no way to mark an account as a "main" account so the most recent one is probably the way to go. Let me know if you have any other suggestions!
    • I am a bit concerned that this could add significant startup time to SteamTinkerLaunch as parsing this file each time could be expensive. That is, if we still store it at startup. We might've only did that for Depressurizer, which is deprecated (#1029). We may only store the userdata path when working with shortcuts, which would negate performance concerns as this is not done every time SteamTinkerLaunch is started.
  2. Allow specifying the Steam User / User ID / both (possibly tricky but we could infer it with regex) in the Add Non-Steam Command, and so also on the UI as well. Usage could be something along these lines, where not specifying at all will result in the Most Recent account being used: steamtinkerlaunch ansg --user 123456.
    • One potential option is to just have the one --user flag, and if it is given and is not the current default User ID, we can parse the Long User ID (the one at the top of each block), the Short User ID (the one you got from the command), the "AccountName" (the name used to log in), and the "PersonaName" (the display name), probably in a loop. Then we can check these in order to see if they were passed into the command (ex: --user GabeN would first check if it's the Long ID, the Short ID, the Account Name, or the Persona Name. Doing this in a loop means we can exit early, and checking if we got the default user's Short ID first could give a speedup as reading these files can be a bit heavy if they are large (although they would probably be as large as something like config.vdf, tens of thousands of lines).

No ETA on implementation, for now as a desparate workaround you could symlink your main account's shortcuts.vdf to the alt account's shortcuts location. That way STL will be writing into the symlink. Then if your alt ever needs to use shortcuts you can just rename the files, basically "juggling" them. It's not ideal but it'll get you up and going until I can take a look at this. Hopefully I can sort the first part soon.


I will rename this issue title and try to get a look at implementing the first part, using the most recently logged in Steam User as the default, this weekend.

WIFIDarthMaul commented 1 month ago

Sure thing! I really appreciate the swift responses and all the help!

The symlink idea was genius, I didn't think of that

On Wed, Jul 10, 2024, 9:27 PM Eamonn Rea @.***> wrote:

Heh, I call it intuition.

The full fix for this is two parts:

  1. Fix how we get the Steam userdata folder, because right now it is not actually basing it on anything other than directory order. I think a reasonable default is the last logged in account. This won't cover all cases, but I think it is a reasonable default, at least better than what we have now. To my understanding there is no way to mark an account as a "main" account so the most recent one is probably the way to go. Let me know if you have any other suggestions! a. I am a bit concerned that this could add significant startup time to SteamTinkerLaunch as parsing this file each time could be expensive. That is, if we still store it at startup. We might've only did that for Depressurizer, which is deprecated (#1029 https://github.com/sonic2kk/steamtinkerlaunch/pull/1029). We may only store the userdata path when working with shortcuts, which would negate performance concerns as this is not done every time SteamTinkerLaunch is started.
  2. Allow specifying the Steam User / User ID / both (possibly tricky but we could infer it with regex) in the Add Non-Steam Command, and so also on the UI as well. Usage could be something along these lines, where not specifying at all will result in the Most Recent account being used: steamtinkerlaunch ansg --user 123456. a. One potential option is to just have the one --user flag, and if it is given and is not the current default User ID, we can parse the Long User ID (the one at the top of each block), the Short User ID (the one you got from the command), the "AccountName" (the name used to log in), and the "PersonaName" (the display name), probably in a loop. Then we can check these in order to see if they were passed into the command (ex: --user GabeN would first check if it's the Long ID, the Short ID, the Account Name, or the Persona Name. Doing this in a loop means we can exit early, and checking if we got the default user's Short ID first could give a speedup as reading these files can be a bit heavy if they are large (although they would probably be as large as something like config.vdf, tens of thousands of lines).

No ETA on implementation, for now as a desparate workaround you could symlink your main account's shortcuts.vdf to the alt account's shortcuts location. That way STL will be writing into the symlink. Then if your alt ever needs to use shortcuts you can just rename the files, basically "juggling" them. It's not ideal but it'll get you up and going until I can take a look at this. Hopefully I can sort the first part soon.

I will rename this issue title and try to get a look at implementing the first part, using the most recently logged in Steam User as the default, this weekend.

— Reply to this email directly, view it on GitHub https://github.com/sonic2kk/steamtinkerlaunch/issues/1140#issuecomment-2221867014, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJBOJ3J4KOUPNNFALW5QFTTZLXUQLAVCNFSM6AAAAABKV2AGFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRRHA3DOMBRGQ . You are receiving this because you authored the thread.Message ID: @.***>

sonic2kk commented 1 month ago

Very interestingly it seems you can add a STEAMUSERID=123456 to the global config file (at ~/.config/steamtinkerlaunch/global.conf, you can put it at the bottom of the file) and use the Short UserID as the value, and SteamTinkerLaunch will use this as the user directory. I don't believe this is exposed on the UI but that is another possible workaround if the symlinking ever fails.

Relevant code: https://github.com/sonic2kk/steamtinkerlaunch/blob/b48b32f91d15de5ef35b1749ef6f35d999bb7916/steamtinkerlaunch#L647-L649

I took a quick look today and it seems there is no way currently to loop through single sections of a VDF file so we'll probably have to do that. We can get values from specific sections, and get single sections, but nothing for generic sections from a browse of the codebase. But I'll try to work on this tomorrow.

sonic2kk commented 1 month ago

This took significant brainpower to figure out a way I was happy with resolving this, due to some limitations of what we can store and parse in Bash given a lack of data structures. I came up with an approach that I'm happy with on the use-mostrecent-loginuser-userdatadir.

That branch right now has code to allow us to use the MostRecent logged in Steam User's userdata folder instead of picking a random one.

I would've preferred a more generic way to parse through and get the blocks from a VDF file, but there is no generic way to do this. So instead we just get the entire "users" section of the loginusers.vdf file and then parse out known patterns matching \t"[0-9]+" which matches every line with one indent that has a series of 1 or more numbers in double quotes, i.e. like "432435490" (which is a known pattern that our loginusers.vdf block start with inside of the "users" block. We can do this because we know the structure of loginusers.vdf. Sadly I didn't figure out a good way to do this generically and to know the difference between a block and a value, so this is a semi-hardcoded solution for loginusers.vdf based on the known structure of the file.

I have some a good amount of testing on this over the last few hours, it works reasonably well for me, but I need to do a bit more testing and then I will open up a PR. I don't think it'll get merged tonight unless I'm feeling brave, so sorry about the delay on this as I did say I wanted to resolve this over the weekend. Since this is a reasonably significant change to a core piece of functionality of SteamTinkerLaunch I want to make extra sure that this does not break finding the userdata folder. I am fairly sure it won't as if the new logic fails we should just fall back to the old logic, but I want to be as close to sure of that as possible before merging.

Thanks for the patience on this :slightly_smiling_face:

sonic2kk commented 3 weeks ago

This took me way too long to get around to testing fully and merging, sorry about that. But this should finally be merged in and ready with SteamTinkerLaunch v14.0.20240727-1 and above. Now SteamTinkerLaunch will use the last logged in user.

If you switch user accounts without rebooting, you will need to remove /dev/shm/steamtinkerlaunch.

There is still pending work to allow selecting the Steam User when adding Non-Steam Games, but now the default will be the last logged in user.