Script has issues with the 'new' Steam Library that released in 2019

[CorruptComputer]

They moved where tags are stored, so I'm going to have to find that before it works again... If I can't find it, I'll have to contact a Valve dev and see what they say about it. I suspect that the new Library design may be web based, so it may not even be stored locally any more.

[PoorPocketsMcNewHold]

For me, The ProtonDB tags still works flawlessly... Except that it can’t detect free games/removed games from the library. According to a post from one dev who made an automatic SteamGrid python script (Not the SteamGrid old program, or neither the new one)

Yeah, they have a Web API, but to use that you need an API key, and it only works if your profile is public. Also not every game you own is returned by that api, it won't count games removed from steam.

https://www.reddit.com/r/steamgrid/comments/d8q0ff/created_a_script_to_automatically_download/f1dqvb9?utm_source=share&utm_medium=web2x

[CorruptComputer]

Interesting, for me the script still works however when it says it updates a tag the game does not show up in the new one for me. I'll see if I can get in contact with that person and if they would be willing to share how he did it or contribute it here.

[4maself]

Are there any updates on this? The script still doesn't work on the beta on my end. When reverting back to the normal cliënt, however, the tags show up as expected.

[CorruptComputer]

@4maself I haven't made any additional progress yet, for now I would say if this is extremely important for you to have you should refrain from using the new library beta. I sent an email to a Steam developer to see if it is still at all possible.

I looked into the script mentioned above and couldn't find anything useful for this project. None of the files or directories accessed stored anything related to tags.

[CorruptComputer]

Never heard anything back about it, but I did find this which is from a related project: https://github.com/Depressurizer/Depressurizer/issues/165#issuecomment-535678912

To re-import your old categories into the Beta do the following with the Steam Client open and running the Beta:

1. Bring up the run window by holding Windows key and the r key
2. Put the following command into the run window's input box: steam://resetcollections
3. You will get a warning message that any new collections you made will be lost while re-importing your old categories. Selecting OK to continue.

I'll test this, and if it works I'll get it added to the script to xdg-open that link.

[CorruptComputer]

It seems that it DOES work, and provides this nice dialogue:

Not sure how adding new games is going to be handled by this though... But in any case, the existing games can at least have updated categories now. I'll get this commit pushed up.

[j800r]

Not sure if this issue is fully resolved yet (about to pull down latest source for testing) but it just become critical.

The new steam library as of just now has left beta. It's official.

[4maself]

Just tried that workaround you made. Only worked with Firefox for me. Both Opera and Chromium couldn't do anything with it. I'm running Manjaro Gnome if that matters, but if you want any more info you can just mention me.

[CorruptComputer]

You should be able to just run the script. At the end it runs a xdg-open which should automatically forward the event to your Steam client letting it know to do this.

One change which needs to change from before though would be to make sure that steam is open so it can receive the event.

[4maself]

Yeah I saw that it should've done that, but for me it just opened a new Opera window. Steam was left alone. I experimented a bit with the link on the browsers that are currently installed and only Firefox seemed to know what to do with it. When I pasted the link into the browsers manually, Chromium and Firefox just opened a google search tab.

[CorruptComputer]

Hmm, could you give me some more info about your distro and python versions? I would like to look into that, since it shouldn't happen that way.

[4maself]

Sure, I'm running python 3.7.4 on Manjaro release 18.1.2 Gnome version 3.34.1 with kernel 5.3.7-2 If you'd like to know any other info, you know where to find me

[PoorPocketsMcNewHold]

Note that you can also launch that command trought Steam directly, by launching Steam with the -console parameter. As displayed, or via launching it via the terminal.

[jkhsjdhjs]

You should be able to just run the script. At the end it runs a xdg-open which should automatically forward the event to your Steam client letting it know to do this.

For me it also opens my web browser first. However, if I invoke xdg-open steam://resetcollections directly from the command line it opens steam without opening my default browser. It probably behaves like that because you're using the python webbrowser module: https://docs.python.org/3.8/library/webbrowser.html#webbrowser.open

 webbrowser.open(url, new=0, autoraise=True)
    Display url using the default browser.

One change which needs to change from before though would be to make sure that steam is open so it can receive the event.

That would indeed be great!

[CorruptComputer]

According to this comment on StackOverflow (not sure how reliable that is though) webbrowser should use xdg-open for those URL's: https://stackoverflow.com/questions/4216985/call-to-operating-system-to-open-url#comment79568357_4217323

For me it does, I don't have to do anything else but run the script as is and it works fine, opening the dialogue in Steam once its completed. @4maself My Python version is the same, as are most of the other versions you gave. It could just be that its behavior depends on your default browser, and if that's the case I'm not sure if there is anything I can do to change it.

[4maself]

Yeah the problem would probably be there. A workaround to that could be using os.system("xdg-open " + url). That handles the url in the cli instead of in a browser. I got that from a little bit of googling https://stackoverflow.com/questions/52210112/python-xdg-open Ofcourse up to you to use it or not.

[CorruptComputer]

That may be the most reliable option, I'll have to add a check for the platform also just in case someone is running this on Windows or Mac. I can do that tonight.

[jkhsjdhjs]

I just had a conversation about the webbrowser module with _habnabit in #python on freenode.

webbrowser.open() uses an array called _tryorder which contains different browsers. It first tries the first entry, if that doesn't work the second entry and so on: https://github.com/python/cpython/blob/3.7/Lib/webbrowser.py#L76 Every browser is added to _tryorder in the register() function: https://github.com/python/cpython/blob/3.7/Lib/webbrowser.py#L32 The register() function is called here multiple times: https://github.com/python/cpython/blob/3.7/Lib/webbrowser.py#L502 Now usually xdg-open should be added first and thus remain the first entry in _tryorder, but if it adds a browser that is a substring of os_preferred_browser, it is made the first entry in _tryorder (as seen in the register() function). In my case it detects firefox and because firefox is a substring of firefox-nightly.desktop, it is opened with firefox.

So yes, the webbrowser module does use xdg-open, but only if it doesn't detect something else as default browser. And the detection isn't reliable, as seen in my case, since firefoxand firefox-nightly are not the same.

So in your case you never want to open the steam:// URL in a browser (except maybe when xdg-open isn't available). So you could use webbrowser.get("xdg-open") to ensure xdg-open is used, and catch an error if xdg-open isn't available. If xdg-open is available you can use .open() on the returned controller object and webbrowser.open() otherwise.

Or you could build something else on your own, since the webbrowser module is made for opening webbrowsers and not the default application for different types of URLs.

[CorruptComputer]

Ah I just realized I didn't say anything here yesterday. So I did work on this, and changing the code to:

            # Workaround provided by Valve for the new library
            url = "steam://resetcollections"
            try:
                webbrowser.get("xdg-open").open(url)
            except:
                webbrowser.open(url, new=0, autoraise=True)

Didn't seem to work. Is that code what you meant by this?

So in your case you never want to open the steam:// URL in a browser (except maybe when xdg-open isn't available). So you could use webbrowser.get("xdg-open") to ensure xdg-open is used, and catch an error if xdg-open isn't available. If xdg-open is available you can use .open() on the returned controller object and webbrowser.open() otherwise.

[jkhsjdhjs]

Yes, that's what I meant and for me it works:

>> ~/ProtonDB-to-Steam-Library % ./ProtonDB-Tags.py
Steam found at: /home/jkhsjdhjs/.local/share/Steam/userdata
Found user 0: id1   acc1
Found user 1: id2   acc2
Which user number would you like to open? 0
Selected: /home/jkhsjdhjs/.local/share/Steam/userdata/id1/7/remote/sharedconfig.vdf
WARNING: This may clear your current tags on Steam!
Would you like to save sharedconfig.vdf? (y/N)y
 >> ~/ProtonDB-to-Steam-Library % Running Steam on arch rolling 64-bit
STEAM_RUNTIME is disabled by the user

[CorruptComputer]

Just a quick update, I'm still trying to figure out the best way to approach getting new games into the script, as Steam no longer updates the sharedconfig.vdf file with the new Library release. I was thinking of just including a json file of all Steam AppID's with it and categorizing them all. But that seems like it would cause it to take too long to run.

Does anyone have any other suggestions?

[PoorPocketsMcNewHold]

I suggest, using the Steam API to see what package a user have. Either from their API key or credentials (Like ArchiSteamFarm) to have a detailed list of the Steam packages they have activated on their account. Or, Use the Steam API with just the user account ID to see all the PUBLIC packages they own. However, this one would be limited if the user has confidentiality settings set, or for the games, tools and software that doesn't have any store pages, and probably doesn't comes up publicly in the user list of own products.

[Saphareas]

So I found a workaround: Open your ~/.steam/steam/userdata/[user id]/7/remote/sharedconfig.vdf and ~/.steam/steam/userdata/[user id]/config/localconfig.vdf side by side in a text editor. Then copy the "Apps"{...} section from localconfig to sharedconfig. Then run the script as you normally would, but don't launch Steam yet when prompted to do so. Instead set the sharedconfig to read-only. Then continue with launching Steam and running steam://resetcollection.

Before this the script only found and tagged 6 games, but now all my 151 titles are tagged. I don't know if setting the file read-only is necessary, but upon launch Steam would sometimes overwrite parts of it.

[PoorPocketsMcNewHold]

@Saphareas commented on Dec 21, 2020, 1:21 PM GMT+1:

So I found a workaround:
Open your ~/.steam/steam/userdata/[user id]/7/remote/sharedconfig.vdf and ~/.steam/steam/userdata/[user id]/config/localconfig.vdf side by side in a text editor. Then copy the "Apps"{...} section from localconfig to sharedconfig. Then run the script as you normally would, but don't launch Steam yet when prompted to do so. Instead set the sharedconfig to read-only. Then continue with launching Steam and running steam://resetcollection.

Before this the script only found and tagged 6 games, but now all my 151 titles are tagged.
I don't know if setting the file read-only is necessary, but upon launch Steam would sometimes overwrite parts of it.

It seems like it helped, from a ~96.3Ko to ~127,1Ko file. Still far from tagging all of my game however as you can see. Still 2164 out of 2604 remain untagged.

[CorruptComputer]

@Saphareas Great find! Thanks for that info, I'm a bit busy with work and the holidays right now so it may be a couple weeks before I can get around to adding this to the script.

If anyone wants to make a PR for it I would be happy to review and approve it.

I have also planned on changing the script to pull that info from the Steam API as well and just write out anything that was missing, but since starting this new job I've been waaaaay too busy to have time to add that functionality.

[CorruptComputer]

Looks like it might be awhile longer until I am able to work on this, my graphics card died and I sent it off for an RMA. Working off of a Raspberry Pi now, which normally wouldn't be an issue, but Steam doesn't seem to be available for ARM processors so any code I write I'm not actually able to test...

[CorruptComputer]

Alright, I have my PC back now and I'm going to work on this in the next few days. One quick question to everyone, I've created an AWS Lambda function to get the users list of games based off their Steam ID that is present in their localconfig.vdf and was going to be adding this to the script. However I'm not sure how people feel about this approach, I figured it would be easier than having everyone make their own API keys and then plug them into the script, but does anyone have any concerns about this?

The code for the lambda function is here: https://github.com/CorruptComputer/ProtonDB-Tags-GetSteamGamesForUser I haven't added any error checking at all, so I wouldn't call it quite ready, but you can use it with this URL: https://49qvinqbbl.execute-api.us-east-1.amazonaws.com/prod/GetSteamGamesForUser?steam_id=[steam_id]

The Steam ID is located in the file path as well as this file (replace the numbers with your own): ../Steam/userdata/85729560/config/localconfig.vdf

"UserLocalConfigStore"
{
    -- some lines here are removed for sake of keeping it short --
    "friends"
    {
        "PersonaName"       "CorruptComputer"
        "85729560"     <--- This is your Steam ID, note it is the same as in the file path
        {
            "name"      "CorruptComputer"
        }

[CorruptComputer]

Also something to note about using the Steam API, your Steam profile privacy settings will need to be public. This would be the case even if people were using their own API keys in a local script, its just not something that you can really get around when using the API.

[CartoonFan]

Has there been any progress on this? Just curious.

[CorruptComputer]

Has there been any progress on this? Just curious.

Not really, I kinda stopped after it seemed like this was a futile effort.

[Technetium1]

@CorruptComputer Maybe it's time to contact someone at Valve and see what they would propose... Maybe smcv / TTimo . There are a lot of people at Valve on GitHub. kisak-valve seems to be someone that often assigns tags to issues, and delegates them to others.

[CorruptComputer]

The latest dev branch should do quite a bit to help with this. The new --fetch-games option will fetch your games list from the Steam API. You will need a Steam API key for this, however the script will walk you through where to get one.

If anyone here has a bit of time, would you be willing to try this option? Be warned that there is a chance that this will clear the current tags from the 'new' games that the script finds from the API, since Valve doesn't provide any way to read the current tags your game has.

EDIT: Just remembered I did a quick proof of concept before using an AWS Lambda function, however this implementation in the dev branch is not using that at all. Your Steam API key and Steam ID will ONLY be used to communicate directly with the Steam API, and will be stored locally on your PC. To read more about this change, please see #45

[CartoonFan]

Seems like the latest changes work for me :+1:

[CorruptComputer]

Talked to someone about this who is maintaining a similar project, they apparently just directly edit the leveldb the same way that the Steam client would, rather than use the steam://resetcollections work around. This allows them to add collections without removing what is already there.

Going to explore this possibility, and they are going to share their source code with me so I can get an idea of the schema of this DB. Hopefully this would make the integration with the newer client seamless :crossed_fingers:

[CorruptComputer]

They still haven't gotten back to me on the schema for that DB file yet, so I haven't had a chance to look into this. If anyone knows any other projects (preferably license compatible with this project) that do the same thing I would be happy to look at those for reference instead, if not I guess we just have to wait until they get back to me.

[jtfen86]

@CorruptComputer Perhaps this project has what you need? https://github.com/solsticegamestudios/GModCEFCodecFix Commit I think that is relevant: https://github.com/solsticegamestudios/GModCEFCodecFix/commit/bcfd65663e4722de3c2a78bfd02b8aa0a695e93a Was looking for your thing, I installed it from AUR and get this error:

[jfenton@jfpc1 ~]$ ProtonDB-Tags --help
Traceback (most recent call last):
  File "/sbin/ProtonDB-Tags", line 8, in <module>
    from Utils.CacheManager import CacheManager
ModuleNotFoundError: No module named 'Utils'
[jfenton@jfpc1 ~]$ ProtonDB-Tags 
Traceback (most recent call last):
  File "/sbin/ProtonDB-Tags", line 8, in <module>
    from Utils.CacheManager import CacheManager
ModuleNotFoundError: No module named 'Utils'

[CorruptComputer]

@jtfen86

@CorruptComputer Perhaps this project has what you need? https://github.com/solsticegamestudios/GModCEFCodecFix Commit I think that is relevant: solsticegamestudios/GModCEFCodecFix@bcfd656

I took a look at that project, and I don't think that is doing the same thing we need here.

Was looking for your thing, I installed it from AUR and get this error:

[jfenton@jfpc1 ~]$ ProtonDB-Tags --help
Traceback (most recent call last):
  File "/sbin/ProtonDB-Tags", line 8, in <module>
    from Utils.CacheManager import CacheManager
ModuleNotFoundError: No module named 'Utils'
[jfenton@jfpc1 ~]$ ProtonDB-Tags 
Traceback (most recent call last):
  File "/sbin/ProtonDB-Tags", line 8, in <module>
    from Utils.CacheManager import CacheManager
ModuleNotFoundError: No module named 'Utils'

Please report this issue to the AUR package maintainer, looks like they aren't including the scripts in the Utils folder with the package. This is not something I manage at all, so I can't help much with it.