[Bug] HBL Modifies shortcuts.vdf incorrectly

[cbartondock]

Before posting this issue, make sure you've read the Issues section in the Readme and the Flatpak/Steam-Deck guide.

Describe the bug I'm the dev for Steam ROM Manager. I've noticed that several users that also use HBL are experiencing bugs in SRM.

The reason is the way that the steam file shortcuts.vdf gets modified by HBL. The structure of shortcuts.vdf (once converted to a JSON) should be

{
  "shortcuts": [
    0: {...}, //app1
    ...
  ]
}

But instead it's

{
  "shortcuts": {
    "": {...}
  }
}

Since SRM expects an array under "shortcuts", it breaks when trying to push things to it. Perhaps HBL is handling this incorrectly when the user is only adding a single game to Steam?

To Reproduce Not sure because I don't use HBL. But basically, add (maybe just one) game to steam with HBL then try to use SRM (or other similar tools).

Expected behavior Shortcuts.vdf readable and writeable by SRM after being modified by HBL.

Screenshots Here is the jsonified version of a shortcuts.vdf file that has been modified by HBL, as seen in SRM:

Details

• Platform: Any
• OS: Any
• Heroic Version: Not sure
• Bash Launcher Version: Not sure

[cbartondock]

Here's another photo where the difference between manually/SRM added shortcuts (top) and HBL added shortcuts (bottom) is more apparent:

[Nocccer]

Maybe we use all the same library. Saw that this is available in python and node: https://github.com/Lagg/steamodd/

Maybe there is another better library

[cbartondock]

That's not a bad idea. SRM currently uses https://github.com/node-steam/vdf, don't think there is a python equivalent. But really as long as that library parses and vdfifies correctly, it should work.

[Nocccer]

Found the python library through this node library: https://github.com/rjackson/node-vdf

They say it is a rewrite of it in node.

[cbartondock]

Oh nice that's perfect then.

[Nocccer]

So actually does not matter which library, but we need to use one. Thanks for this tipp.

[redromnon]

I feel it'll be better to use the Python library that @Nocccer found since it parses the vdf similar to the one used by SRM. I haven't dug deep into it since I'm currently busy.

[Nocccer]

You just need the vdf serializer. There are also other packages in python.

[Nocccer]

@cbartondock actually i'm confused that you use less entries for each game. I thought you need to declare all of the possible entries: https://github.com/CorporalQuesadilla/Steam-Shortcut-Manager/wiki/Steam-Shortcuts-Documentation#shortcut-entry-structure

[cbartondock]

No I don't think you do - steam is clever enough to just ignore ones that aren't set. - For the sake of Accuracy™ it would probably be best to do so, but as far as I know it causes no issues to set only the ones SRM sets.

[Nocccer]

Actually i try the vdf node library you use and it does not work for writing ? It writes plain json into vdf file ? From my understanding writeFileSync(shortcutsFile, stringify({ shortcuts: {} })) should do resolve to �shortcuts� in shortcuts.vdf and not to shortcuts: {}

Actually if i wanna parse the shortcuts.vdf steam creates which contains �shortcuts� the parse fails.

I'm so confused right now. Because steam creates by default a shortcuts.vdf binary file, which is totally different than other steam .vdf files. HBL was currently modifing the binary shortcuts.vdf as steam does it. SRM things it is a none binary .vdf file and treat like the other .vdf files from steam. So what is the correct way. Because by default steam creates the binary file, which is not readable by node-steam/vdf library.

@cbartondock Removing the shortcuts.vdf and add a game manually to steam, you will see how the shortcuts.vdf looks like. For me it is binary and exactly the same as described here: https://github.com/CorporalQuesadilla/Steam-Shortcut-Manager/wiki/Steam-Shortcuts-Documentation#shortcut-entry-structure

[cbartondock]

Whoops! That's because that's the wrong library I gave you. Refer to https://github.com/SteamGridDB/steam-rom-manager/blob/master/src/lib/vdf-shortcuts-file.ts

What we are using is steam-shortcuts-editor.

Sorry for the bad steer!

[cbartondock]

For the record node-steam/vdf is what we use to parse / edit 760/screenshots.vdf

[Nocccer]

Ok thanks so i will see if there is a python library for that aswell

[Nocccer]

@redromnon maybe this works ?

https://pypi.org/project/steamclient/

[cbartondock]

It looks right to me! If you want to send me a shortcuts.vdf that has been created /edited by it I can test to see if it has the correct structure when converted to json

[redromnon]

@redromnon maybe this works ?

https://pypi.org/project/steamclient/

This should do it. Thanks a lot @Nocccer for this!

[cbartondock]

When you fix this be sure to have some mechanism in place to deal with already corrupted files, otherwise you will most likely see the same errors as SRM due to the fact that an array of shortcuts is expected by the library.

[Nocccer]

Maybe there is a way we could test if a created shortcuts.vdf is not corrupted after changes and if so we could bring up the old shortcuts.vdf

[cbartondock]

That would be best - to migrate it to the correct format rather than just delete it and create a new one.

[Nocccer]

But actually there is no way without starting steam

[cbartondock]

No there is. Check if the field shortcuts exists and if it is an array. If it is an object migrate things over to the array format.

[cbartondock]

Ah but your library doesn't output a JSON? Just specific entries etc?

[Nocccer]

For me i can use your check. But sometimes you miss something and your check pass but steam complains about shortcuts.vdf. Would be good to have something to see if steam can read it.

I adding a feature to heroic to add non-steam games. Heroic is written in typescript

[cbartondock]

Yeah I mean unless you want to run the steam executable and somehow record if it has an error (no idea how you would do that) the best you can do is try to guarantee the file conforms to your expectations - if steam changes things on their end (which they usually don't but obviously could) then it will break and we'll have to fix it.

That's the game of cat and mouse dozennn and I have been playing for years now lol.

[redromnon]

@Nocccer Unfortunately https://pypi.org/project/steamclient/ doesn't work because it needs winreg (Windows Registry Access) that is not available on Linux.

[Nocccer]

@redromnon you can use the class from the project and write a function were you can pass the filepath. I looked over the source code and it looked pretty easy todo

[redromnon]

Okay, thanks. I'll look into that. How about this - https://github.com/CorporalQuesadilla/Steam-Shortcut-Manager? A bit older but simple.

[Nocccer]

Also possible. What fits better for you.

Maybe you need the shortcut.py and make it a function inside your source code and make it python3 compatible

[Nocccer]

@cbartondock can you test for me this shortcuts.vdf. I created it manually with steam. It seems the steam-shortcut-editor does not create correct json. shortcuts.zip

it creates this. Mess up the AppName entry.

{
        shortcuts: [
          {
            appid: 1857929199,
            '���\x01AppName': undefined,
            iscord: undefined,
            Exe: '"/usr/share/discord/Discord"',
            StartDir: '"/usr/share/discord/"',
            icon: '',
            ShortcutPath: '/usr/share/applications/discord.desktop',
            LaunchOptions: '',
            IsHidden: false,
            AllowDesktopConfig: true,
            AllowOverlay: true,
            OpenVR: false,
            Devkit: false,
            DevkitGameID: '',
            DevkitOverrideAppID: false,
            LastPlayTime: 1970-01-01T00:00:00.000Z,
            FlatpakAppID: '',
            tags: []
          }
        ]
      }

[cbartondock]

The one you uploaded is correct. If you want you can use this to test them: https://replit.com/@ChrisDock/Shortcuts-VDF-Read#index.js

[Nocccer]

Weird. The appid is different than mine. So probably something on my end. Are you sure the number in steam can be negative ?

Can you do a simple fs.readFile and parseBuffer and log the parsed output ? For me it gives still the error.

I'm really confused. We using typescript and i wrote types for the steam-shortcut-editor. But still without types i get this weird output. Also if i copy your code you shared above. Can you try it with Typescript? I think Typescript Buffer is maybe not the same as JavaScript Buffer ?

[redromnon]

@cbartondock I have a theory. Does SRM expect the srno and the appid in the vdf when converting into json format? HBL currently doesn't include those two and this may be the issue.

I'll try to fix this. Otherwise I will have to make use of the project I last mentioned.

[cbartondock]

SRM puts the app id in there for the sake of accuracy and I think because very rarely steam uses it, but no it doesn't depend on it already being there. We compute the app id directly from the location of the executable and the title name.

The main thing that is causing an error in SRM is the fact that it expects an array under shortcuts.

[redromnon]

I've noticed that once HBL adds the games to shortcuts.vdf, the user needs to manually add another shortcut so that Steam corrects the syntax and then probably move forward to use SRM. Obviously this is an inconvenient workaround.

[Nocccer]

Found this library for reading vdf in python. https://github.com/ValvePython/vdf

[redromnon]

I had already come across this library but it is not used to read shortcuts.vdf. It is intended to be used for devs for the KeyValue format in Valve's Source Engine. I tried but it doesn't work and gives UnicodeDecode Error.

[redromnon]

I'm currently working on (close to) fixing this issue in HBL itself rather than depending on a library.

[Nocccer]

@redromnon This library (https://github.com/ValvePython/vdf) can be used for shortcuts.vdf. You just used it wrong. You need to read and use the binary functions.

Working example i wrote: https://replit.com/@Nocccer/SteamShortcut#main.py

[redromnon]

Yep, that's a stupid mistake on my end. Thanks for correcting me. I'd used 'r' instead of 'rb' for reading and didn't realize this at all. No wonder why I was getting the UnicodeDecode error.

[redromnon]

@Nocccer and @cbartondock Can you both please confirm if this shortcuts.zip file generated by HBL from scratch works? If it doesn't work for SRM, I'll have to use the library Nocccer suggested.

I don't know how to parse and dump the appid as it remains 0 until a new shortcut is added by Steam. But what's important is that the correct vdf format will be preserved and should (hopefully) be no issues with SRM.

[Nocccer]

I would suggest the library. If i'm done in heroic i can copy and rewrite it in python.

@redromnon Looks good here: https://replit.com/@Nocccer/SteamShortcut#main.py

[Nocccer]

Found something interesting. If you manually adapt the AppName directly in shortcuts.vdf and a appid is defined. The parser will fail somehow with the error i had above.

@cbartondock do you have a shortcuts.vdf with alot of games (20). Writing test atm and would be good to have one which was created by SRM.

[redromnon]

I would suggest the library. If i'm done in heroic i can copy and rewrite it in python.

@redromnon Looks good here: https://replit.com/@Nocccer/SteamShortcut#main.py

Great! Looks like there'll be no further problems with SRM and even the function you are working on for Heroic. I don't have to depend on any library anymore. I just to had carefully revise the format given by Valve here - https://developer.valvesoftware.com/wiki/Add_Non-Steam_Game

[redromnon]

For further testing, this workflow build can be used to test HBL with the fix.

[cbartondock]

Sorry I was travelling, but great work. Shall I close the issue here and over at SRM?

[redromnon]

No issues. Yeah, you can close it.