Non-Steam games are not listed in the games list.

[marco-calautti]

When clicking the games list button, I can only see steam games, but not the external ones added as shortcut.

Steps to reproduce the behavior:

1. Open ProtonUp-Qt
2. Select steam (Flatpak in my case)
3. Click on "Show game list"

Expected behavior
All games, both steam and non-steam should be shown

Having also non-steam games listed would allow to automatically bump the proton version for all games configured in Steam.

[sonic2kk]

ProtonUp-Qt reads the ~/.local/share/Steam/appcache/appinfo.vdf file to determine which games are owned and installed I believe. However from what I can see from the Non-Steam Games that I added, these entries are not written out to the appinfo file.

I believe they are written elsewhere if memory serve right from working on STL (logically, Steam has to store it somewhere I would guess, how would it remember the given name, compat tool, etc?), but I cannot find or remember where at the moment.

Couple of small notes around this issue that come to mind:

• Not all Non-Steam Games will have a compatibility tool, but not all compatibility tools without a compatibility tool will be native.
• Some tools on Steam Deck like EmuDeck add a lot of Non-Steam shortcuts, which could make the list quite long. Maybe an option to filter Non-Steam games would be helpful.

I'll do a bit of investigation into this and see if I can figure out a way to implement it :-)

[sonic2kk]

Information about shortcuts is stored at ~/.local/share/Steam/config/<steam_userid>/config/shortcuts.vdf. I think the Steam library can parse this but I haven't checked yet.

There's a different problem though. This file, as far as I know, doesn't store information about the compatibility tool used for the file. When updating the compatibility tool used in Steam, this file is not modified.

Another couple of files are modified, including a licensecache file. The more interesting file is the localconfig.vdf file. This file is fairly big, and contains lots of misc information about the Steam installation - your friends list, some CDN information, and a list of entries under Software.

I scrolled through the file and tried to find somewhere that had some information about the compatibility tool, but even under the Software entries I could really only see information about cloud saves and some last played time.

[marco-calautti]

From what I know, the compatibility tool mappings are all stored in a vdf inside the config folder, also for the non-steam games. I regularly manually change it to bump the proton version for all my non-steam games.

I am not at my machine now, but I will check once I can.

[DavidoTek]

Thanks for the investigation.

Having also non-steam games listed would allow to automatically bump the proton version for all games configured in Steam.

That an interesting idea which might be worth adding to ProtonUp-Qt.

ProtonUp-Qt reads the ~/.local/share/Steam/appcache/appinfo.vdf file to determine which games are owned and installed I believe. However from what I can see from the Non-Steam Games that I added, these entries are not written out to the appinfo file.

Installed Steam games are read from ~/.local/share/Steam/config/libraryfolders.vdf and information about these games are read from appinfo.vdf which contains information (only?) about owned games.

Information about shortcuts is stored at ~/.local/share/Steam/config//config/shortcuts.vdf. I think the Steam library can parse this but I haven't checked yet.

Yes, ~/.local/share/Steam/userdata/<steam_userid>/config/shortcuts.vdf seems to store the shortcuts. Not sure what file format it is using (some sort of binary vdf format). PS off-topic: Might be interesting for https://github.com/DavidoTek/ProtonUp-Qt/issues/48 as we can add a shortcut for ProtonUp-Qt to Steam this way.

From what I know, the compatibility tool mappings are all stored in a vdf inside the config folder, also for the non-steam games

Similar to Steam games, compatibility tool information for Non-Steam games is stored in ~/.local/share/Steam/config/config.vdf. Although I'm not sure where it gets the ID from...

[sonic2kk]

The compatibility tool mappings are stored in ~/.local/share/Steam/config/config.vdf, under the CompatToolMappings key.

The problem is that this file works based on AppID. Non-Steam Games do not really have "standard" AppIDs, for example an entry may look like this:

"3187211279"
{
    "name"      "Proton-5.8-GE-2-MF"
    "config"        ""
    "priority"      "250"
}

The question is now, well how does Steam know/generate the AppIDs for Non-Steam Games? Well, I suffered trying to figure this out in the past and didn't get anywhere :sweat_smile:

Several months ago I spent weeks banging my head against the wall trying to figure out this specific problem. In the end I figured out how it's meant to be generating the AppIDs, but I could not replicate the same AppIDs that Steam was coming up with for each game. Also, if you remove and re-add a Non-Steam Game to Steam, the AppID changes even if the paths and so on are the same.

The basic idea is that Steam generates the AppID from the game name and game path (I believe the path to the folder, with the executable at the end). It then uses some sort of CRC algorithm to generate this. You can see this StackOverflow answer even lists a solution in Python. However in my testing, I could not get this to match the AppIDs that Steam generated. Here is the Python code for reference:

from pathlib import Path
import binascii

def get_steam_shortcut_id(exe, appname):
    """Get id for non-steam shortcut.

    get_steam_shortcut_id(str, str) -> str
    """
    unique_id = ''.join([exe, appname])
    id_int = binascii.crc32(str.encode(unique_id)) | 0x80000000
    return id_int

Steam ROM Manager uses what to my eye looks like an identical CRC algorithm to generate the AppID used by Steam. It's written in TypeScript, but it looks pretty much the same to me as what the Python code does and the theory described in the StackOverflow answer. The interesting thing to note about Steam ROM Manager is that it is used and reliably works including on Steam Deck, where it is used in conjunction with EmuDeck to batch-add ROMs as Non-Steam Game shortcuts and set their game artwork (which requires knowing the AppID). However when I tried to run the code in the linked generate-app-id.ts in an isolated TypeScript environment, I was not getting the AppIDs generated by Steam. The TypeScript code and the Python code were generating the same AppIDs, but it was not the same one used by Steam. Here is the TypeScript code for reference:

// 'crc' is a separate npm library: https://www.npmjs.com/package/crc
// There are a couple but I verified that this is the correct one based on the package.json: https://github.com/SteamGridDB/steam-rom-manager/blob/master/package.json#L98
import * as crc from 'crc';
function generatePreliminaryId(exe: string, appname: string) {
  const key = exe + appname;
  const top = BigInt(crc.crc32(key)) | BigInt(0x80000000);
  return BigInt(top) << BigInt(32) | BigInt(0x02000000);

}

// Used for Big Picture Grids
export function generateAppId(exe: string, appname: string) {
  return String(generatePreliminaryId(exe, appname));
}

// Used for all other Grids
export function generateShortAppId(exe: string, appname: string) {
  return String(generatePreliminaryId(exe, appname) >> BigInt(32));
}

// Used as appid in shortcuts.vdf
export function generateShortcutId(exe: string, appname: string) {
  return Number((generatePreliminaryId(exe, appname) >> BigInt(32)) - BigInt(0x100000000));
}

// Convert from AppId to ShortAppId
export function shortenAppId(longId: string) {
  return String(BigInt(longId) >> BigInt(32));
}

// Convert from ShortAppId to AppId
export function lengthenAppId(shortId: string) {
  return String(BigInt(shortId) << BigInt(32) | BigInt(0x02000000));
}

---

So that's how the AppID for Non-Steam Games is supposed to be generated. If we could find a way of getting this to work, probably half the battle would be one. However, it is important to note two things:

• The shortcuts.vdf file does not have any AppID reference. It does not store the Non-Steam AppID
• The config.vdf only stores compatibility tool mappings based on AppID. It does not store any other information.

We would need some kind of way to link the Non-Steam AppID and game information in the shortcuts file together.

[marco-calautti]

I am quite sure steamgrid uses this CRC algorithm to find the games IDs. See it here: https://github.com/boppreh/steamgrid/blob/36f069ef2a9bb0b1f0f0af7e17fa1d8bd68e627c/games.go#L105

[sonic2kk]

Indeed, that's what I found out over my weeks of investigation in the past, but this never seemed to generate the correct AppID for me, so I am not sure. The CRC algorithm you linked is also referenced in the StackOverflow answer I linked.

[marco-calautti]

I use steamgrid regularly to download the artwork for my non-steam games, and has always worked.

[sonic2kk]

It works for me as well, as I also mentioned:

The interesting thing to note about Steam ROM Manager is that it is used and reliably works including on Steam Deck, where it is used in conjunction with EmuDeck to batch-add ROMs as Non-Steam Game shortcuts and set their game artwork (which requires knowing the AppID).

I am not an SRM developer and haven't looked too deeply at the code, but perhaps they generate (or fetch?) the Non-Steam AppID differently now. Perhaps something changed and the functions I listed are no longer used. That would be the only way I could explain why SRM works and the code running standalone does not.

It's possible to test the SRM code on something like Replit without having to install/setup TypeScript locally. this is what I did in the past (https://github.com/sonic2kk/steamtinkerlaunch/issues/550#issuecomment-1264026429) and that was how I figured out that none of the sample code online for generating the Non-Steam AppID seemed to generate anything consistent with what Steam gives back.

There are a couple of ways to check the Non-Steam AppID. You could set some game artwork and then go to ~/local/share/Steam/userdata/65217330/config/grid/ - The filename for the art you set will be prefixed with the AppID. If it's a game using Proton, the AppID will be used for the compatdata name at ~/.local/share/Steam/steamapps/compatdata. You could also set a specific compatibility tool only for that Non-Steam Game and then verify it in the config.vdf.

[DavidoTek]

In the end I figured out how it's meant to be generating the AppIDs, but I could not replicate the same AppIDs that Steam was coming up with for each game

That is very strange. I was able to generate the app id using the Go code, but Python did not.

crc32.ChecksumIEEE([]byte("\"firefox\"Firefox")))|0x80000000 produces 4065111499 (correct), while binascii.crc32(str.encode('"firefox"Firefox')) produces 3418141247 (wrong).

Maybe binascii isn't using the "IEEE CRC32 Checksum"? Hmm. zlib.crc32 gives the same wrong result.

[sonic2kk]

Just to be sure, remember to include the 0x80000000 in the Python code as well (it wasn't mentioned in the paste you did but maybe you still have it in 😅)

Interestingly, most of the (incorrect) AppIDs I remember generating also started with 34XX(...)

[DavidoTek]

I have noticed the Go code differentiates between a gameID (correct one) and a LegacyID (wrong, using CRC32).

gameID := fmt.Sprint(binary.LittleEndian.Uint32(gameGroups[1]))   // -> 4065111499 (correct)
LegacyID := uint64(crc32.ChecksumIEEE(uniqueName)) | 0x80000000   // -> 3418141247 (wrong)

I did a little test and extracted b'\xcb\xadL\xf2' from between appid\x00 and \x01AppName\x00 in the vdf file. By running int.from_bytes(b'\xcb\xadL\xf2', byteorder='little'), i get the correct result.

So we probably need to implement a parser similar to the Go one.

It seems like they changed the format: https://github.com/boppreh/steamgrid/commit/9c8788db4f04613ecfb3e8fb36a0af02395e4593

---

The new format seems to work like this. Maybe this is some known format:

Data for a Non-Steam game look like this: \x01<key>\x00<value>\x00 (some use \x02 instead of \x01). Example: \x01AppName\x00Firefox\x00

Each game entry looks something like this \x00<entry number, e.g. 0,1,2...>\x00<entries>\x08\x08

[marco-calautti]

I know there are proper binary vdf parsers for python you can use, and thus you can avoid reading the raw binary. See for example: https://pypi.org/project/vdf/

[DavidoTek]

Oh right, seems like the vdf library supports vdf.binary_loads. That's good as we're already using it for some other stuff.

But for some reason it does not correctly parse the app id...

Show parsed `shortcuts.vdf` file

`vdf.binary_loads( open('shortcuts.vdf', 'rb').read() )` outputs: ```json { "shortcuts":{ "0":{ "appid":-229855797, /* wrong, should be 4065111499 */ "AppName":"Firefox", "Exe":"\"firefox\"", "StartDir":"\"./\"", "icon":"", "ShortcutPath":"/usr/share/applications/firefox.desktop", "LaunchOptions":"", "IsHidden":0, "AllowDesktopConfig":1, "AllowOverlay":1, "OpenVR":0, "Devkit":0, "DevkitGameID":"", "DevkitOverrideAppID":0, "LastPlayTime":0, "FlatpakAppID":"", "tags":{} } } } ```

[marco-calautti]

That's maybe because it is reading the number as a signed integer, and thus interprets it as a negative number.

Indeed, that is a 32 bit integer that the library interprets as signed. If you convert it to unsigned integer, you will get the right number.

[marco-calautti]

[marco-calautti]

Are we sure we really need to compute the crc? Steam did it already for us, right? In the shortcuts.vdf we already have the appid, which we can look for in the compatibility tool mapping inside config.vdf, right?

[sonic2kk]

Sorry to bump this but I figured this was as good of a place as any to give an update, in case it's relevant in the future, or if anyone from the future stumbles across this thread and is curious. To be clear, this only applies to Non-Steam AppIDs. Regular game AppIDs are explicitly generated and set by Valve, and I don't believe there's any way to change them (or any meaningful reason behind doing it, since they're always fixed and can be fetched).

While Steam does calculate an AppID for Non-Steam Games, as long as you write out a 32bit signed integer to the shortcuts.vdf as 4byte little-endian hex, this can be anything.

I tested this using the Python VDF library. It was possible to do the following:

1. Load the shortcuts.vdf with the VDF library
2. Change nonsteamgames['shortcuts']['0']['appid'] to have any amount of numbers changed (you could change one or the entire number).
3. Write this back out to shortcuts.vdf using the VDF library, as this will convert from decimal to hex for you (can't remember what it was now, probably just storing binary_dumps into a variable and writing it out with file.write(vdf_path, 'wb')) -- Now a custom AppID is set for the shortcut.
   • Note that this hex conversion isn't simply as straightforward as hex(my_signed_int), because this will return a hexidecimal string like 0x16c66563 which is not 4byte little-endian, which the Non-Steam AppID must be written out as. I used this tool to convert from my unsigned integer to 4byte little-endian hex, and then compared the hex value it gave me with the one from binary_dumps.
   • I then took this hexidecimal and convert it to a 32bit signed integer, and then also converted that hexidecimal straight from hex to the 32bit unsigned integer, and compared it the signed int converted to unsigned (see next point) and they matched.
4. To get the unsigned AppID (such as the one used for shortcuts), convert the negative AppID from an 32bit signed integer to unsigned. You can do this with my_signed_int & 0xFFFFFFFF.
   • It's also possible to convert by going from shortcuts.vdf hexidecimal -> signed AppID -> unsigned AppID, though I'm not sure of an implementation.
5. You now have both AppIDs that Steam uses; the 32bit signed integer (which is stored in hex in shortcuts.vdf), and the 32bit unsigned integer (which is used for Non-Steam Game artwork, compatdata folder names, etc).

So in theory, you could just generate a random 32bit signed integer, and write that out to the shortcuts file. However, this means there's a possibility of AppID conflicts. I haven't actually tested what happens in a case like this, but I assume Steam will try to resolve it if there are two shortcuts with the same AppID.

This conflict is where I think the idea of the CRC comes in, and why tools like Steam-ROM-Manager generate a CRC using the appname and Exe fields as keys. In doing this you can pretty confidently generate AppIDs that won't conflict -- That won't stop Steam itself from having this issue though, if you add Non-Steam Games through the Steam client.

When you add a Non-Steam Game through Steam it sets the AppID then and there, but this AppID is different each time. If you remove and re-add the exact same game, it will have a different AppID. Without checking the shortcuts.vdf, one easy way to check this is to make sure it's a game running via Proton and get Steam to generate a compatdata, the compatdata directory will have a different name each time.

This could be a bug in the Steam Client, but I think it's just however Steam generates the AppIDs that makes it different each time. In other words, there's no strict way that Steam generates AppIDs, and we also don't have to follow this.

---

Once I figured all this out, I was able to write an implementation in Bash (sonic2kk/steamtinkerlaunch#902) for creating the 32bit signed integer (using the appname and exe as seeds), and from there generating and writing out the value in hexidecimal, as well as getting the 32bit unsigned integer used for the AppID everywhere else on Steam. I was able to test that this worked by logging that unsigned AppID, setting some game artwork in ~/.local/share/Steam/userdata/<>/config/grid, and when I opened Steam, that artwork was properly selected.

This approach isn't quite perfect as it doesn't checking for conflicting AppIDs, so the AppID generated here could theoretically conflict with the ones Steam creates. It also works a bit differently than how Steam ROM Manager might work, because I used the shuf utility to create the random number using a seed. This means if the seed is the same, the AppID will always be the same. This was just a nifty side-effect I happened across, but I wanted to mention that explicitly in case someone else wants to write an implementation that works this way, too.

I currently haven't written a way (in Bash) to parse the AppID from the shortcuts.vdf file, but I'm hoping to do that eventually.

---

I've spent over a year trying to figure this out and it was this straightforward all along: so long as the AppID is written out in the correct format, the AppID can be manually set by us to anything. Talk about taking the long way around! Hopefully my documented findings here will bring some resolution to this whole fiasco.

[DavidoTek]

Thanks for the extensive description.

I think it is really great to have such documentation in case anyone else needs to implement such a functionality. Some functionality like this were really a pain to get working by reverse engineering as no one documented this stuff before. :rocket: