Rework Steam library import

[shakeyourbunny]

Check for existing issue Could not find this issue in the ticket system.

Describe the bug Steam version of these games are not imported:

• Alone in the Dark 1
• Alone in the Dark 2
• Alone in the Dark 3

These games gave errors on import, but they are in the library?

• Yume Nikki
• Velvet Assassin
• XCOM: Enemy Unknown
• Mordheim: City of the damned
• Sword Coast Legends
• Orcs Must Die! Unchained ( this f2p game was removed and not playable anymore, this is ok)
• Trillion
• The Walking Dead: A New Frontier

To Reproduce

• activate Steam Library Importer.
• Steam Library Options: (x) import installed games, (x) connect account, (x) import uninstalled games, authenticate user, (x) provide API key for "private account"
• update game library -> Steam Steam games will be imported, but at least not Alone in the Dark 1, 2, 3

Screenshots Steam versions of Alone in the Dark in Playnite https://s1.imagebanana.com/file/200511/BslUXHWu.PNG

Alone in the Dark in my Steam client https://s1.imagebanana.com/file/200511/UBNveZW0.PNG

Diagnostics ID 2020-05-11 08:34:04.5872|INFO:Importing games from Steam plugin. 2020-05-11 08:34:04.5872|INFO:Steam game Das Geisterschiff Remixed is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.5872|INFO:Steam game Enderal - Forgotten Stories OST is not properly installed or it's a soundtrack, skipping. ... other soundtrack skipping removed ... 2020-05-11 08:34:04.5983|INFO:Steam game Velvet Assassin is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.5983|INFO:Steam game XCOM: Enemy Unknown is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.6223|INFO:Steam game Mordheim: City of the Damned is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.6323|INFO:Steam game Sword Coast Legends is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.6464|INFO:Steam game Orcs Must Die! Unchained is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.6584|INFO:Steam game Trillion is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.6584|INFO:Steam game The Walking Dead: A New Frontier is not properly installed or it's a soundtrack, skipping. 2020-05-11 08:34:04.6835|INFO:Steam game Yume Nikki is not properly installed or it's a soundtrack, skipping. .... 2020-05-11 08:34:04.7075|DEBUG:Found 457 installed Steam games. 2020-05-11 08:34:04.7075|DEBUG:Downloading string content from https://api.steampowered.com/IPlayerService/GetOwnedGames/v0001/?key=redacted&include_appinfo=1&include_played_free_games=1&format=json&steamid=76561198055778112 using System.Text.UTF8Encoding encoding. 2020-05-11 08:34:06.1173|DEBUG:Found 6631 library Steam games.

It is funny that my Steam client says that I have 7255 games (see https://s1.imagebanana.com/file/200511/UBNveZW0.PNG), Playnite finds 6631/6680 games (see https://s2.imagebanana.com/file/200511/06oL0YkF.PNG ) and my online profile states that I have licensed 4622 games (https://steamcommunity.com/id/shakeyourbunny). I also checked the latter games tab where it could not locate the three Alone in the Dark games I do have.

Alone in the Dark 1: appid 548090 https://s1.imagebanana.com/file/200511/nXUvl9Wg.PNG Alone in the Dark 2: appid 548890 https://s1.imagebanana.com/file/200511/ehWueJaU.PNG Alone in the Dark 3: appid 548900 https://s2.imagebanana.com/file/200511/PtvbNEo9.PNG

SteamDB entry for the series: https://steamdb.info/search/?a=app&q=Alone+in+the+Dark&type=1&category=0

[shakeyourbunny]

Uploaded diagnostics, ID: 8eb68930-fc4d-4044-bbc0-f01b0b756473

[shakeyourbunny]

For reference, SteamDB seems to correctly see that these games are owned by my account.

I dunno HOW SteamDB does this, especially when I just obtained the two games "Football Girls: Dream Team" (appid 862600) and "Speedy Girls - Dream Team" (894900). Both games are not detected by Playnite with a rescan or fresh library either.

Alas, this got me digging with the Steam Web API (https://steamapi.xpaw.me/) and steamcmd.

Steamcmd does list Alone in the Dark 1,2,3 AND the both Speedy Girls games with "steamcmd +login myusername +licenses_print +quit".

Alone in the Dark 1,2,3 License packageID 135448:

• State : Active( flags 512 ) - Purchased : Wed Dec 26 14:28:18 2018 in "AT", PayPal
• Apps : 259170, 548090, 548890, 548900, (4 in total)
• Depots : 259171, 548091, 548891, 548892, 548893, 548894, 548895, 548901, 548902, 548903, 548904, (11 in total) AppId 259170: https://steamdb.info/app/259170/

Football Girls: Dream Team License packageID 271840:

• State : Active( flags 0 ) - Purchased : Thu May 21 12:29:40 2020 in "AT", CD Key
• Apps : 862660, (1 in total)
• Depots : 862661, 862662, (2 in total)

Speedy Girls - Dream Team License packageID 285829:

• State : Active( flags 0 ) - Purchased : Sun Mar 15 18:44:28 2020 in "AT", CD Key
• Apps : 894900, (1 in total)
• Depots : 894901, 894902, (2 in total)

as for the error'ed appids, I picked "Velvet Assassin" License packageID 1596:

• State : Active( flags 512 ) - Purchased : Sat Jan 5 14:40:57 2013 in "AT", Credit Card
• Apps : 16720, (1 in total)
• Depots : 16721, 16722, (2 in total)

I did some digging with steamcmd and the Steam WebAPI (see https://steamapi.xpaw.me), after I obtained the two games, but both on the public games list these games are not visible nor the WebAPI Call https://api.steampowered.com/IPlayerService/GetOwnedGames/v0001/?key=myapikey&include_appinfo=1&include_played_free_games=1&format=json&steamid=mysteamid which PlayNite uses. It does not matter if include_played_free_games and/or include_free_sub, they do not influence the result.

My guess what SteamDB does to properly get all games which the account really has, is building the game list by parsing all licensed package IDs properly. The call https://steamapi.xpaw.me/#ISteamGameOAuth/GetPackageInfo may or may not plays a role, but I'm not sure, because this call did not accept my steam api token.

[shakeyourbunny]

Oh, and SteamDB says that I have 6400 games :D (what gives..) on my profile page there :)

[shakeyourbunny]

Well, after sifting through xPaws Github page, I discovered the Steam License Parser ( https://github.com/xPaw/SteamLicenseParser online demo: https://lab.xpaw.me/license-parser.html ).

The recipe for obtaining all the games of a user on Steam should be:

• provide a valid steam login (token)
• get list of licensed package IDs (sub(scriptions)
• go through each sub and extract the app ids into a list.
• (optional?) use IPlayerService / GetOwnedGames and do a check if something is missing.

[JosefNemec]

I'm sorry for late response.

If Steam web API doesn't report those games, then it's a bug in the API, you should report it to Valve. All other solutions that return list of Steam licenses also return DLCs and other non-game items, which means checking each and every license (more API calls) to see whether the item is a game or not. That would slow down import hugely.

For now you can workaround this by installing those games, letting Playnite import them, and then uninstalling them. They will stay imported in Playnite after that.

As far as those other games, can you try repairing their installation? The message from logs means that Steam didn't properly create necessary files we use to import installed games.

[psychonic]

You don't really need many extra calls.

If you pull the full list of owned apps from a logged in Steam web session @ https://store.steampowered.com/dynamicstore/userdata, then you can reference the local appinfo.vdf to check the app type (game, etc.) for each. SteamDB has a .NET library for parsing it, as it's not quite the same as Valve's normal binary keyvalues format. https://github.com/SteamDatabase/SteamAppInfo

[JosefNemec]

Using appinfo.vdf is not safe. The very first Playnite version used it and there are multiple issues with it. Mainly the file not being accessible or being invalid because of Steam writing to it at the time we access it.

[JosefNemec]

The safest way would be probably creating some Steam app info cache on our backend that would have that info and could be shared between users, which would minimize Steam API calls a lot.

[shakeyourbunny]

https://store.steampowered.com/dynamicstore/userdata

That was the piece I was missing... I could not make head or tails out of it where in the world the package list was coming from.

The safest way would be probably creating some Steam app info cache on our backend.

I think this would ease the strain on the Steam API calls, but it'd suggest it would have some sort of TTL (time to live) until an entry is re-fetched. This could be done locally though too, my private Steam stuff also just dump the stuff into JSON based cache files (sqlite or other may be a better solution).

PS: I'm a Python guy, doing my .. own stuff for archving purposes, sorry that I am not really versed in .NET.

[JosefNemec]

I'll probably look into it during Playnite 9 development and switch to using that userdata endpoint. The cache would realistically only need to keep game ID and whether it's a game or something else. General metadata download can be done locally, like we already do it.

My guess is that this would also fix issue JosefNemec/Playnite#910

[shakeyourbunny]

A suggestion (or a workaround) for obtaining the userdata:

Add an option to import cookies from a valid Steam login from a browser. With this information, PlayNite has a valid login into Steam and access to https://store.steampowered.com/dynamicstore/userdata

[TopHatHipster]

Add in the following games not being imported:

Sokpop S09: King Pins Sokpop S09: skelets

The Steam versions didn't import but the itch.io versions did.

[JosefNemec]

The issue with "Profiles Features Limited" games has been fixed in 8.9 update. I don't know if it's still worth redoing Steam integration since that was the main reason I wanted to do it in the first place.

[JosefNemec]

Closing since most issue here have been fixed on Steam's side (like feature limited games) and the rest is imo too niche to be worth redoing the whole auth, but more importantly not worth not using official API anymore, which gives us more stability in long term.

There's always a workaround where you can temporarily install a game, let it import and then uninstall it, which will leave it imported in Playnite.