search

sonic2kk / steamtinkerlaunch

Linux wrapper tool for use with the Steam client for custom launch options and 3rd party programs
GNU General Public License v3.0
2.15k stars 73 forks source link

SteamGridDB support for Non-Steam games #906

Closed CartoonFan closed 1 year ago

CartoonFan commented 1 year ago

System Information

Feature Description

I'm sure you've got a ton of stuff on your plate at this point :laughing: , but after glancing at #738 and the recent commits, I was wondering if it'd be possible to have SteamGridDB grab (and assign) the game art for a specific game.

My current workflow for adding Non-Steam games is:

  1. Install through Lutris and and use the built-in setting to add it to Steam
  2. Use steamgrid to add the artwork to Steam
  3. Rename the game to its Steam ID (where possible) to take advantage of Steam Input

Probably doesn't seem like a lot, but when you've got a bunch of Non-Steam games, it gets to be tedious fairly quickly. So...I was wondering if Steam Tinker Launch could maybe simplify the process somehow :thinking:

Thanks!

sonic2kk commented 1 year ago

I like this idea. It crossed my mind before, I think I mentioned somewhere around #757 about having some sort of integration for Non-Steam Games (or at least investigated this without noting it anywhere for some reason).

It is possible to download artwork from Steam Grid for a game somewhere in the game menu / global menu, but I'm not sure if this works for Non-Steam Games, and I believe it's for the artwork you see when launching a game through SteamTinkerLaunch.

I suppose then it should be possible to fetch artwork. The question becomes how to implement this. We can't have as much of a dynamic UI like some other tools have, but I'll have a think and see what I can come up with in the future.

Probably doesn't seem like a lot, but when you've got a bunch of Non-Steam games, it gets to be tedious fairly quickly.

I can certainly see how this would be a pain. For emulation, Steam-ROM-Manager is pretty much magic, so I'd like a feature like this somehow.

Useful idea, no idea on an implementation strategy yet, but should be doable once the UIX portion is figured out (yeah, Yad hinders the UIX in general, but I still do my best to make new features as least painful as possible :wink:).

CartoonFan commented 1 year ago

Useful idea, no idea on an implementation strategy yet, but should be doable once the UIX portion is figured out (yeah, Yad hinders the UIX in general, but I still do my best to make new features as least painful as possible 😉).

That's about as much as I can ask for :smile:

Thanks for the consideration :purple_heart: :pray:

I can certainly see how this would be a pain. For emulation, Steam-ROM-Manager is pretty much magic, so I'd like a feature like this somehow.

I looked it up after the initial version of this comment, and apparently it works for more than just emulation! :astonished:

I'll look a bit more into it to see if I can use it for anything. Thanks for the advice! :+1:

sonic2kk commented 1 year ago

It's still something I'd like to include in STL, even just as a personal challenge of sorts. They got the AppID thing down and I spent well over a year trying to figure out how they did it and, well, I finally did and I did it in Bash too (and even released it independently from SteamTinkerLaunch as a reference implementation for anyone else interested).

No ETA on when this will come, I have a couple of improvements I want to make to configuring+cleaning up Non-Steam Games which will require some time investigating methods of interacting with text-based VDF files. I also want to improve some of the logic around how we write out to the binary shortcuts.vdf file to make it a bit cleaner and more flexible. And of course, there are some SpecialK+ReShade improvements to do :-) Things have been busier around here than they have been in a while so lots of fun things coming!

I'll be doing my best to implement these features when I have time, in the order of the ones I am most motivated for. I appreciate the feature suggestion though and I do try to get around to things eventually (even if it can take several months...) so hope it's ok if it may take a while!

CartoonFan commented 1 year ago

It's still something I'd like to include in STL, even just as a personal challenge of sorts. They got the AppID thing down and I spent well over a year trying to figure out how they did it and, well, I finally did and I did it in Bash too (and even released it independently from SteamTinkerLaunch as a reference implementation for anyone else interested).

Congrats! :partying_face:

No ETA on when this will come, I have a couple of improvements I want to make to configuring+cleaning up Non-Steam Games which will require some time investigating methods of interacting with text-based VDF files. I also want to improve some of the logic around how we write out to the binary shortcuts.vdf file to make it a bit cleaner and more flexible. And of course, there are some SpecialK+ReShade improvements to do :-)

Of course :grin:

Things have been busier around here than they have been in a while so lots of fun things coming!

Looks like you've been busy :eyes:

STL's had some really nice developments recently, so the near future's looking promising :grin:

I'll be doing my best to implement these features when I have time, in the order of the ones I am most motivated for. I appreciate the feature suggestion though and I do try to get around to things eventually (even if it can take several months...) so hope it's ok if it may take a while!

I don't know if there's anything I can do this time to help things along, so...I might just have to wait :sweat_smile:

sonic2kk commented 1 year ago

So thinking about this some more, STL already has support for SteamGridDB, though it works based on AppID. As far as I understand, this just downloads a bunch of grids from SteamGridDB. You can pass your own API token by generating one and adding it to the global config file either directly by modifying the SGDBAPIKEY variable, or by entering it on the Global Menu.

Since this works by AppID, we would need a way to allow searching by game name. We would also need to be pretty strict, narrowing down to only exact name matches (for example, searching for Touhou 6 should only return those matches). The issue is, expecting users to be strict with the game name might be a bit much.

The next problem even if we iron the above out, is how to integrate it with Add Non-Steam Game. Perhaps we could add a button to download the grids without closing the dialog, then we can add in the dialogue description where to find the game grids in the STL folder structure.

I am also not sure how to integrate this properly into the commandline usage, it probably isn't really possible in any meaningful way.

The problem with all of this is, at least to me, it's not that much easier than just browsing SteamGridDB and manually downloading the files. I think there may be a better way to solve this issue, or at least something we can do in furtherance of this goal of adding artwork to Non-Steam Games with lower friction.

With that, going in a completely different direction, there is a potential alternative to solve this issue. One solution could be to read game artwork from the game EXE folder based on name

This approach does have some issues though:

I also considered a way of trying to set this based on image dimensions, but that would be too unreliable I think. I don't believe SteamGrid enforces a standard size, and a user may want artwork that doesn't fall into the standard dimensions that Valve recommend.

This direction, I think, is potentially the better way to resolve this. The user searches for and saves the images they want and then puts them in with the game EXE, and by default STL picks it up automagically by filename. The user still has to go to StemGridDB to get the artwork but that means they get to review and choose it. Plus if they're adding from the commandline, the command becomes much simpler as it gets picked up. From the UI, it also saves them some clicks, and they don't have to worry about what they name the EXE (such as using a dash instead of a tilde for Touhou 6 ~ Embodiment of Scarlet Devil).

If you have any thoughts on this feature or a different way to solve this problem please suggest away! Though keep in mind Yad is pretty limited when it comes to any kind of dynamic UI behaviour, but suggesting things could help us discuss and come up with another solution that can work, so please feel free when you can to suggest whatever way you think would be good to implement this feature :-)

CartoonFan commented 1 year ago

I apologize if any of this is off-point; I really don't know much about how SteamGridDB works.

Since this works by AppID, we would need a way to allow searching by game name.

It's still hard for me to believe that SteamGridDB can't just pull the name from the Steam AppID. Like, it's right there in the name 😜

We would also need to be pretty strict, narrowing down to only exact name matches (for example, searching for Touhou 6 should only return those matches). The issue is, expecting users to be strict with the game name might be a bit much.

We can't just use a drop-down box? For example, on SteamGridDB's website, typing in "steins" grabs six different Steins;Gate games (are there more? IDK :shrug:), with Steins;Gate being the first one. Typing "Touhou 6" also has Touhou 6 as the first suggested game,so it seems like SGDB can work with somewhat fuzzy searches.

Still, I feel like you know more about this than I do, so I can't really argue too hard here :thinking:

This direction, I think, is potentially the better way to resolve this. The user searches for and saves the images they want and then puts them in with the game EXE, and by default STL picks it up automagically by filename. The user still has to go to StemGridDB to get the artwork but that means they get to review and choose it. Plus if they're adding from the commandline, the command becomes much simpler as it gets picked up. From the UI, it also saves them some clicks, and they don't have to worry about what they name the EXE (such as using a dash instead of a tilde for Touhou 6 ~ Embodiment of Scarlet Devil).

I like this. Still rather tedious, but it was nice to do that for my Steam games, so I imagine it'd be the same here :+1:

If you have any thoughts on this feature or a different way to solve this problem please suggest away! Though keep in mind Yad is pretty limited when it comes to any kind of dynamic UI behaviour, but suggesting things could help us discuss and come up with another solution that can work, so please feel free when you can to suggest whatever way you think would be good to implement this feature :-)

Is there a way to...not use the exact game name from the UI? Like, kind of a way to somehow feed a search-friendly title into SteamGridDB, then (perhaps) have checkboxes for which assets the user would like to keep (or replace?) That way, they could still have the ability to choose which assets they want, and let SteamGridDB choose the others.

Hopefully my suggestions will improve as I better understand all this :sweat_smile:

sonic2kk commented 1 year ago

It's still hard for me to believe that SteamGridDB can't just pull the name from the Steam AppID. Like, it's right there in the name

Non-Steam Games don't have AppIDs the same way Steam games do, that AppID is created. When you add the game to Steam manually, Steam generates a different AppID each time. For STL we have a fixed way of generating the AppID. That's why SteamGrid can't search by AppID for Non-Steam Games.

For Non-Steam Games, the AppID is generated when the information is written to shortcuts.vdf, which is where the AppID is written out to, it's a field in this file.

For games on Steam, the AppID is constant and used to identify the game such as on the Store Page.

We can't just use a drop-down box? For example, on SteamGridDB's website, typing in "steins" grabs six different Steins;Gate games (are there more? IDK 🤷), with Steins;Gate being the first one. Typing "Touhou 6" also has Touhou 6 as the first suggested game,so it seems like SGDB can work with somewhat fuzzy searches.

We can't have dynamic UI behaviour like that in SteamTinkerLaunch. All the information has to be fetched before the dialog box is loaded. We don't have a way to know what game a user will look for before opening the Add Non-Steam Game dialog, and we can't have dynamic behaviour on the UI because Yad doesn't allow for that. We can't for example have a drop-down appear while the user is typing and allow them to select the game name.

We also have no useful way of letting users preview the images, the closest is a thumbnail list view which scales and stretches the images down a lot, which isn't useful for things like heros.

Artwork would have to be fetched by game name (because that's the only way to know what game we're dealing with, AppID is generated and not fixed) when the user clicks the button to add the shortcut to Steam, or when they run the ansg command. We can't have dynamic behaviour to let them search for images within SteamTinkerLaunch.

Sadly the UIX for this is probably not feasible but I hope I explained the limitations properly.

Is there a way to...not use the exact game name from the UI? Like, kind of a way to somehow feed a search-friendly title into SteamGridDB, then (perhaps) have checkboxes for which assets the user would like to keep (or replace?) That way, they could still have the ability to choose which assets they want, and let SteamGridDB choose the others.

The problem is once the user clicks "Add" we can only fetch and add the artwork once. So we need an exact match to ensure the user gets the artwork they want. True, it's on them if they don't get the one they want, but to me it reduces the usefulness of the feature.

I also have some concerns of essentially adding a "frontend" for using SteamGridDB, as a third-party tool, even if it was feasible to do something like this in Yad (which it really isn't to be honest).

So really, we can't have any way to fetch X amount of artworks from SteamGridDB and pick the desired artwork in SteamTinkerLaunch. That's just sadly not possible with Yad.

But, we could add the functionality to just fetch the first best-match artwork, and gate that behind a checkbox like "fetch artwork from SteamGridDB". The user doesn't get to choose the artwork but the feature I mentioned of trying to auto add artwork from the game EXE folder would be a lower-friction way to set the artwork for users who want that extra flexibility. Doing this may be better than nothing?

CartoonFan commented 1 year ago

But, we could add the functionality to just fetch the first best-match artwork, and gate that behind a checkbox like "fetch artwork from SteamGridDB". The user doesn't get to choose the artwork but the feature I mentioned of trying to auto add artwork from the game EXE folder would be a lower-friction way to set the artwork for users who want that extra flexibility. Doing this may be better than nothing?

I'm still OK with this :grin:

Just to explore the idea a bit more--since I'm still not 100% clear on the dynamic/static differences--would it be possible to let a user specify the SteamGridDB ID--rather than the Steam AppID--when adding a game or before loading the UI? If a user knows the exact game on SteamGridDB, there wouldn't be any need for a title search :thinking:

sonic2kk commented 1 year ago

Oh, I didn't realise SteamGridDB had specific game IDs. It makes a ton of sense, I dunno what I thought, I guess I assumed they stored by game name or something and maybe some unique identifier after it.

I would assume this ID is constant, but yeah it would be totally feasible to have an optional SteamGridDB ID for the game to fetch game artwork.

I'm thinking we can work this feature like this:

I think I have a much clearer idea on how to implement this feature request now. Thank you for pointing out the ID thing I probably wouldn't have caught that. I think being able to pass the ID when adding the game and doing a search is more than doable and I like it a lot more than simply guessing.

Again though they won't really be "choosing" the artwork from SteamGridDB but still think this solution is okay :-)

CartoonFan commented 1 year ago

Oh, I didn't realise SteamGridDB had specific game IDs. It makes a ton of sense, I dunno what I thought, I guess I assumed they stored by game name or something and maybe some unique identifier after it.

Yeah, I found this out the hard way when I was trying to rename a batch of games to their Steam IDs (for Steam Input and such). Got a good way through the list before I realized they weren't the right IDs :laughing:

My guess is that SteamGridDB's use of their own IDs is at least partly due to supporting other distribution platforms besides Steam.

To give an example of what I'm seeing with the different IDs:

SteamGridDB's system seems pretty versatile--which is to say, not limited to Steam games--so hopefully their IDs should work for a wide variety of games :crossed_fingers:

I would assume this ID is constant, but yeah it would be totally feasible to have an optional SteamGridDB ID for the game to fetch game artwork.

I hope so :grimacing:

The whole thing would fall apart if they suddenly changed it :sweat_smile:

I'm thinking we can work this feature like this:

  • Have a checkbox something like "Fetch Artwork from SteamGridDB

  • Have an optional textbox to enter the SteamGridDB ID that will only be read if the above checkbox is enabled

  • If the option is enabled but no SteamGridDB ID is passed, try to search based on the game name

    • For commandline usage we will need some consideration around when and how to read the ID/name. I'm thinking we'll only use the ID and fallback to the name if a flag like --use-steamgriddb is passed (so even if the ID is passed from the commandline it'll be ignored if the flag to use SteamGridDB is false)

    • SteamGridDB artwork will take priority over any selected game artwork as well as any artwork in the game EXE folder

      • If, say, SteamGridDB only returns a banner and some boxart, we'll then check if the user selected any artwork. This will then be used as a fallback. If the user selected a logo, then that logo will be a fallback for the one SteamGridDB didn't find
      • finally, for this example, if there was no tenfoot found, we'll check if there's a file in the same folder as the game EXE to use

  • Update text on the commandline help screen and the Add Non-Steam Game UI to inform users when and how to use this feature -- We'll also update the wiki

    • We'll note about the SteamGridDB API key and how to update it both from the UI, textfile, and commandline
    • We'll note that SteamGridDB ID is used primarily and then we fall back to game name
    • finally we'll note that the "best match" artwork will be used. Of course a user can change any artwork they don't like from Steam, or if they want that extra flexibility they can always select the artwork themselves.

Sounds good! :+1:

I think I have a much clearer idea on how to implement this feature request now. Thank you for pointing out the ID thing I probably wouldn't have caught that.

You're welcome! :grin:

I think being able to pass the ID when adding the game and doing a search is more than doable and I like it a lot more than simply guessing.

Nice :sunglasses: :+1:

Again though they won't really be "choosing" the artwork from SteamGridDB but still think this solution is okay :-)

With SGDBoop, one can choose their own heroes/grids/etc. for their Non-Steam games through the Web UI, so they're not completely stuck with whatever SteamGridDB decides. I still think the idea of selecting specific assets and putting them in the game directory for STL to pick up is still a really cool idea, actually :thinking:

Even though we've already covered a lot of ground between these two comments; would it be possible to have STL overwrite the game's name with the Steam ID (where present, and provided by the user)? I'm pretty sure it's needed for Steam to find community input layouts (and possibly other things). It doesn't have to be something that's always on, but the problem is that SteamGridDB searches with the game's name--which is less of an issue with our above changes--but Steam uses the Steam ID, and ignores the game name entirely.

Because SteamGridDB uses the game name (or its own ID for it), I feel like the Steam ID-renaming would be best after SteamGridDB does its work, but I'm not sure if that's possible with the current state of the tools in use :sweat_smile:

That's pretty much it! Even one of these two big changes would be a helpful timesaver :purple_heart: :+1:

sonic2kk commented 1 year ago

Sounds like the path forward for this proposal is an option to fetch the popular grid artwork from SteamGridDB with an optional SteamGridDB ID and a fallback to the entered name. If not provided, we fall back to the EXE name, so we could search on the basename without the extension (the only required field for ansg is the EXE and STL will fill out any other required field). In the case of this feature being off, SteamGridDB being down, or artwork not being found, we'll fall back to selected artwork or, if no artwork is selected, search for artwork in the same folder as the game EXE based on its filename.

Even though we've already covered a lot of ground between these two comments; would it be possible to have STL overwrite the game's name with the Steam ID (where present, and provided by the user)? I'm pretty sure it's needed for Steam to find community input layouts (and possibly other things). It doesn't have to be something that's always on, but the problem is that SteamGridDB searches with the game's name--which is less of an issue with our above changes--but Steam uses the Steam ID, and ignores the game name entirely.

I'm not entirely sure what you mean here. As in if you added a game called "Touhou 6" it would overwrite it with whatever the Non-Steam AppID we generate for it is? For Non-Steam Games, there is no fixed AppID. Each time you add a game to Steam from the Add Non-Steam Game option in the Steam client, the ID changes. For STL, we generate a fixed AppID. There is no option to enter the AppID by the user, and although there could be, it would be a pretty specific feature.

Are you saying you want an option to force the game name to be the AppID? Sorry, I don't really understand the ask here. Both specifying an AppID and overwriting the game name would be possible but I don't understand the reasoning or if this is exactly what you're asking.

CartoonFan commented 1 year ago

I'm not entirely sure what you mean here.

I have a habit of doing that, unfortunately :sweat_smile:

As in if you added a game called "Touhou 6" it would overwrite it with whatever the Non-Steam AppID we generate for it is? For Non-Steam Games, there is no fixed AppID.

Ooh, I didn't know this. Thanks for the clarification :+1:

Each time you add a game to Steam from the Add Non-Steam Game option in the Steam client, the ID changes. For STL, we generate a fixed AppID. There is no option to enter the AppID by the user, and although there could be, it would be a pretty specific feature.

Are you saying you want an option to force the game name to be the AppID? Sorry, I don't really understand the ask here. Both specifying an AppID and overwriting the game name would be possible but I don't understand the reasoning or if this is exactly what you're asking.

My ask was more for games that are both on Steam and another distribution platform. For example, I have a license for Rebel Galaxy (https://www.steamgriddb.com/game/4195) on GOG--but not Steam. If I added Rebel Galaxy to Steam, as a Non-Steam Game, I wanted to know if it was possible to feed STL the actual Steam ID, either with the commandline or in a field in the UI. I'd likely have to supply the Steam ID myself, but I feel like it would reduce the number of steps in the whole process of adding a game :sweat_smile:

If it's still unclear, I can try to give more info :purple_heart: :smile:

EDIT: It wouldn't be applicable to games that aren't natively on Steam. After I posted this comment, I realized that was part of the actual question you asked :laughing:

sonic2kk commented 1 year ago

I think I understand what you mean now, you want to use the Steam ID for a game that was not bought on Steam when added as a Non-Steam game, to take advantage of the controller configurations.

If I added Rebel Galaxy to Steam, as a Non-Steam Game, I wanted to know if it was possible to feed STL the actual Steam ID, either with the commandline or in a field in the UI.

Sadly this isn't really possible for a couple of reasons, but the main one is that Non-Steam AppIDs have to be a specific type of integer. If you've never seen a Non-Steam AppID, they're something like 393475821. On a technical note, these are unsigned 32bit little endian integers. Basically, a negative 32bit integer, and it's stored in a specific way (little endian, or "reverse byte order" in hexadecimal to the shortcuts.vdf file. And Steam will actually verify this, because if the AppID is not in this format, it is not recognised by Steam and will be overwritten whenever Steam reads the shortcuts.vdf file. This is why the Non-Steam AppID generation didn't work before and figuring this out was why it took so long to implement #738.

When adding the game to Steam through the Steam Client, Steam will come up with the AppID itself. When Non-Steam Games are added to Steam, a binary file called shortcuts.vdf is modified. This is located at ~/.local/share/Steam/userdata/<user-id>/config/shortcuts.vdf. This file has various fields but it stores the AppID as well. Steam just writes out to this field when you add the AppID yourself, and any other program that modifies this file can write this out as well. If the AppID is not specified, or if the format is invalid, Steam will overwrite it the next time it reads from the file. If generated properly though, you can really write out any AppID you want to this file so long as it matches the format Steam expects, and to my knowledge it's not feasible to do this with numbers smaller than a certain range (at least in my testing if the number was less than 9 digits, Steam didn't like it). You can even modify the AppID of existing shortcuts if you want, it's not super trivial but it is possible to do it if you generate the hex and use a hex editor to modify the file, or parse it using something like the Python VDF library where you can simply set a shortcut to have any negative 9 digit number you want, and the library will take care of writing it out.

Native Steam games use a different ID, which I'm not sure on the specifics of but I know that it goes up overtime. For example:

In short, Steam expects a specific format of AppID for Non-Steam Games, and this can't really be spoofed to match a Steam native AppID because Steam will overwrite AppIDs that don't match the expected format. There would also be AppID conflicts if the game was ever purchased on Steam in future, and since the Steam native AppID can't be changed, the Non-Steam Game AppID would change, messing up grids, compatdata, and anything else relying on the AppID.

As a small note on the commandline, there isn't any extra functionality on the commandline for ansg. The UI calls the same function as the command from terminal would (link), so it's all the same internally, except the UI adds a frontend to enter the information passed to the commands :-)

I'd likely have to supply the Steam ID myself, but I feel like it would reduce the number of steps in the whole process of adding a game

There may be other solutions for the controller configuration issue, you might be able to ask around and see what others are doing, but I don't think the AppID solution is possible. I'm sure others have solved it some other way though, maybe there's a site that collates them. Maybe SteamDB has a way to browse controller configs, and you could download them and put them into an internal Steam folder named after the Non-Steam AppID. You can find the Non-Steam AppID right after you add it in the SteamTinkerLaunch log, and it's also mentioned when you launch the game via SteamTinkerLaunch.

CartoonFan commented 1 year ago

Hmm...it's been a while, but I think just replacing the game name with the Steam ID (from SteamGridDB, ProtonDB, etc.) was enough for Steam to find the controller configurations for the game, even in cases where I also had the game on Steam (Saints Row 2 and 3, IIRC), so it's not a huge thing if I have to do it manually. Just trying to streamline the process as much as I can :laughing:

Assuming that SteamGridDB thing works out, though; renaming the game is a relatively small, one-time time/effort cost, especially if it only applies to games that are also on Steam.

Hopefully I'm not making things more confusing :sweat_smile:

Thanks for the in-depth explanation, BTW :+1: Didn't really understand all of it completely, but I understand it a bit better now.

sonic2kk commented 1 year ago

Hmm...it's been a while, but I think just replacing the game name with the Steam ID

As in instead of having the game name as, say, "Saints Row: The Third", it would be named "55230"? There may have been some confusion then, I was referring to the internal AppID for the game.

To do this STL would have to try and get the AppID for the game, likely from an API call, and if the name wasn't exact (even if it was missing the colon from "Saints Row: The Third", or "Steins Gate" instead of "STEINS;GATE") it would likely fail. We'd also have to manage cases where the game ID isn't found.

I think I get where you're coming from, but I think for most users, not having the game "named" and having it with the AppID is relatively niche. You could always put the AppID in the name field if you want to accomplish this, that way you don't have to rename anything.

Controller configs are stored in ~/.local/share/Steam/steamapps/common/Steam Controller Configs. This has subfolders named either after the AppID of the game, or the Non-Steam Game Name, all lowercase and alphanumeric (so "The Legend of Zelda: Twilight Princess HD" becomes "the legend of zelda twilight princess hd", all lowercase and no colon). I guess then by naming the Non-Steam Game after the AppID, that's how it finds the controller configs, because it searches using the AppID in the name.

If you managed to get the controller config you wanted, or if you already had the config for the Steam copy, you could manually add it this way. For example creating a folder called "saints row the third" and then copying the controller config from the Steam copy stored within folder name 55230. You shouldn't have to rename it if you're using the same controller for each game.

You might also be able to find controller configs online, there may be a site that collates and shares them. Likewise, if the game is a popular Non-Steam Game, community controller configs should already be available if you name your game specifically however other people playing this game as a shortcut are :-)

I think this is a bit out-of-scope for STL, as it's somewhat specific and can be solved by naming the game as the AppID, which is pretty straightforward to get (steamtinkerlaunch gi "game name", SteamDB, ProtonDB, the STL config file name, ProtonUp-Qt's game lists, and the Steam Store Page link which is visible from the Steam Client, are just some ways to get the AppID).

CartoonFan commented 1 year ago

Hmm...it's been a while, but I think just replacing the game name with the Steam ID

As in instead of having the game name as, say, "Saints Row: The Third", it would be named "55230"?

Yeah, that's what I meant. Coincidentally, Saints Row: The Third was a game that actually required me to use a Non-Steam copy :laughing:

After some point in time, Valve's CEG made it simply not work (https://github.com/ValveSoftware/Proton/issues/2228), so I had to use GOG's copy instead.

There may have been some confusion then, I was referring to the internal AppID for the game.

So there's more than one AppID associated with a game...I'm learning so much here! :grin:

To do this STL would have to try and get the AppID for the game, likely from an API call, and if the name wasn't exact (even if it was missing the colon from "Saints Row: The Third", or "Steins Gate" instead of "STEINS;GATE") it would likely fail. We'd also have to manage cases where the game ID isn't found.

I think I get where you're coming from, but I think for most users, not having the game "named" and having it with the AppID is relatively niche. You could always put the AppID in the name field if you want to accomplish this, that way you don't have to rename anything.

That's my plan. My main concern was that SteamGridDB might get confused (since it normally searches with the game name), but with the upcoming improvements--specifically, searching SteamGridDB with the SGDB ID--hopefully everything will work together without issue :crossed_fingers:

Controller configs are stored in ~/.local/share/Steam/steamapps/common/Steam Controller Configs. This has subfolders named either after the AppID of the game, or the Non-Steam Game Name, all lowercase and alphanumeric (so "The Legend of Zelda: Twilight Princess HD" becomes "the legend of zelda twilight princess hd", all lowercase and no colon). I guess then by naming the Non-Steam Game after the AppID, that's how it finds the controller configs, because it searches using the AppID in the name.

If you managed to get the controller config you wanted, or if you already had the config for the Steam copy, you could manually add it this way. For example creating a folder called "saints row the third" and then copying the controller config from the Steam copy stored within folder name 55230. You shouldn't have to rename it if you're using the same controller for each game.

You might also be able to find controller configs online, there may be a site that collates and shares them. Likewise, if the game is a popular Non-Steam Game, community controller configs should already be available if you name your game specifically however other people playing this game as a shortcut are :-)

Hmm...I see :thinking:

I'll have to keep this in mind the next time I add a Non-Steam Game :+1:

I think this is a bit out-of-scope for STL, as it's somewhat specific and can be solved by naming the game as the AppID, which is pretty straightforward to get (steamtinkerlaunch gi "game name", SteamDB, ProtonDB, the STL config file name, ProtonUp-Qt's game lists, and the Steam Store Page link which is visible from the Steam Client, are just some ways to get the AppID).

I feel like a good chunk of my suggestions are out-of-scope for this project :laughing:

Still, as we both mentioned; at worst, it's just a simple rename, so it's not a terribly big deal to do for certain games.

sonic2kk commented 1 year ago

So there's more than one AppID associated with a game...I'm learning so much here!

Sorry I'll clarify. There's only one. The AppID is the numbers that represent the game internally to Steam, and it doesn't depend on the game name after it has been added. The AppID is something like "52230", not "Saints Row: The Third". However, the AppIDs for Steam native and Non-Steam games are different.

Non-Steam Games have, primarily the following fields:

The AppID is generated when a game is added to shortcuts.vdf. Different programs do this differently, but Steam-ROM-Manager and STL generate it based on the Game Name and Exe fields. Once added, this AppID cannot be changed under normal circumstances (you can edit with a hex editor and such though). But the game name can be changed. The Steam ID, or "App ID" is not the same as the Game Name. I hope that helps clarify.

specifically, searching SteamGridDB with the SGDB ID--hopefully everything will work together without issue

Yes, the user will enter the SteamGridDB AppID and STL will use the stored API key to search for the most popular artworks it can find. Since this is SteamGridDB's AppID and is entirely independent to Steam, there shouldn't be any problems :smile:

I feel like a good chunk of my suggestions are out-of-scope for this project

Hm, I thought the vast majority of your suggestions so far have been useful (#894 got a lot of discussion and most of it got implemented to my understanding?). Getting game artwork from SteamGridDB is also feasible, as suggested here. If you'd like more explanation, I can explain more on why I don't think having an option to try and guess the Steam AppID for a Non-Steam game's name is super critical, but it essentially boils down to what I've already said: This is very straightforward to do manually, it's just a copy and paste, and the use-case is probably lacking broad appeal (people usually want the game name to be something "human readable").

Adding an option to change the default game name when the AppID can just be specified manually seems a bit much to me. If there were no way to set the game name I would agree we can add it, but it feels like a lot to have an option to set a specific game name, when you can just copy and paste this value yourself. Trying to automate it would be unreliable and the user can just fetch this themselves when adding a game. Plus, not everyone uses controllers for their games. I never used a controller with Saints Row The Third, for example. It seems like a very specific use-case that is very straightforward to do "manually" (i.e. just entering a specific game name for this specific use-case). There aren't a great deal of steps to automate, but the few that there are would be hard to do reliably.

Not to mention, for SteamGridDB artwork, the user will have to get the SteamGrid ID themselves, so it's not too unreasonable to expect them to enter the game AppID for this one use-case as well. At that point, why wouldn't we have an option to guess the SteamGrid ID? The same reasoning for why we wouldn't do that applies here as well, it's not reliable enough or useful enough to warrant all the fetching for information users can provide themselves pretty easily. It's really not that much of a step to enter the AppID manually as the game name, since this use-case is pretty niche. Not to mention that this isn't even applicable to all games, as there are games without community controller configs to begin with.

I think it's a bit unfair to say a lot of your suggestions are dismissed as out-of-scope. This feature is just entering the game AppID as the game name, I don't see why this needs a dedicated option which is why I don't think it needs to be added. If someone wants it badly enough in the future they could open a PR, but it's not something I think is so crucial it needs its own checkbox or flag to automate. At that point, we would open the door to all sorts of options, like taking in a Steam AppID and setting the game name based on that. It becomes a bit much in my mind to try and account for all of these use-cases in a tinker utility, where these options can be easily done with a copypaste. And for commandline usage, a Bash script could be written to take in a list of AppIDs perhaps from a CSV and name the game that way.

It is a lot simpler to do this manually than it would be to not only implement this feature, but it would be a lot more reliable to do it manually too, as it is just copy and pasting from the Steam Store Page URL or wherever else one gets their AppIDs, it's not exactly hidden information. I think the user should be responsible for naming the game the way they want. It's a lot easier for a human to get the AppID for a game than a Bash script.

CartoonFan commented 1 year ago

So there's more than one AppID associated with a game...I'm learning so much here!

Sorry I'll clarify. There's only one. The AppID is the numbers that represent the game internally to Steam, and it doesn't depend on the game name after it has been added. The AppID is something like "52230", not "Saints Row: The Third". However, the AppIDs for Steam native and Non-Steam games are different.

Non-Steam Games have, primarily the following fields:

  • AppID (ex: 582230) - 32bit unsigned little endian integer

  • Game Name (ex: Saints Row: The Third) - String

  • Exe (ex: ~/Games/GOG/Saints Row The Third/sr3.exe) - Path to the game executable

  • StartIn (ex: ~/Games/GOG/Saints Row The Third/pfx/drive_c/users/emil/My Documents/Saints Row The Third) - The working directory for the executable - generally defaults to dirname of Exe (Steam runs chdir after it starts a process including for Non-Steam Games, you can see this if you launch Steam from terminal and start a game, it's part of the start command)

The AppID is generated when a game is added to shortcuts.vdf. Different programs do this differently, but Steam-ROM-Manager and STL generate it based on the Game Name and Exe fields. Once added, this AppID cannot be changed under normal circumstances (you can edit with a hex editor and such though). But the game name can be changed. The Steam ID, or "App ID" is not the same as the Game Name. I hope that helps clarify.

It does :+1:

specifically, searching SteamGridDB with the SGDB ID--hopefully everything will work together without issue

Yes, the user will enter the SteamGridDB AppID and STL will use the stored API key to search for the most popular artworks it can find. Since this is SteamGridDB's AppID and is entirely independent to Steam, there shouldn't be any problems 😄

That's my main concern solved, then :smile:

I feel like a good chunk of my suggestions are out-of-scope for this project

Hm, I thought the vast majority of your suggestions so far have been useful (#894 got a lot of discussion and most of it got implemented to my understanding?).

I wasn't sure if I was on a lucky streak or not :eyes:

Generally, I tend to ask you if STL can do something, and I rely on you to tell me if it's in-scope or not. With so many intersecting pieces, it's hard for me to tell where the scope of this project begins and ends :sweat_smile:

Getting game artwork from SteamGridDB is also feasible, as suggested here. If you'd like more explanation, I can explain more on why I don't think having an option to try and guess the Steam AppID for a Non-Steam game's name is super critical, but it essentially boils down to what I've already said: This is very straightforward to do manually, it's just a copy and paste, and the use-case is probably lacking broad appeal (people usually want the game name to be something "human readable").

As do I :laughing:

Adding an option to change the default game name when the AppID can just be specified manually seems a bit much to me. If there were no way to set the game name I would agree we can add it, but it feels like a lot to have an option to set a specific game name, when you can just copy and paste this value yourself. Trying to automate it would be unreliable and the user can just fetch this themselves when adding a game.

I mean, if it's easier and more reliable to do manually, that's quite all right.

Plus, not everyone uses controllers for their games. I never used a controller with Saints Row The Third, for example.

That's incredible :eyes:

Basically, I'll use a controller with almost anything that's not a point-and-click game. Besides my keyboard becoming difficult to type with recently (the keys lost their lubricant, and are more difficult to press now), controllers are generally easier on my hands and wrists, so they've become more of a priority for me.

To elaborate on my reasons for wanting SteamGridDB for Non-Steam Games:

I've been using Big Picture for a while now--due to the reasons above--and adding Non-Steam Lutris games to Steam with the STL gives me a grid with just the name of the game, and a low-res hero that just looks...bad. The game name is also on the Play screen after I select a game, which also doesn't look great.

On top of all that, as we've discussed here, I had been renaming the game name with the Steam ID, to take advantage of Steam Input; but that just puts "55230" (using Saints Row: The Third as an example) everywhere within Steam. Not only does this also look "bad", IMO, but I have to constantly remember that "55230" is "Saints Row: The Third"; which is quite frustrating when one is casually scrolling through their list of games, as I'm sure you can imagine.

Even besides these two issues, the games just "stand out" more than I'd like, especially if I'm viewing a collection with my Steam and Non-Steam games together.

SteamGridDB resolves the visual problems listed here, bringing high-quality assets, and removing the game name from where the hero is, making everything look just right.

As for the renaming itself:

Pros:

Cons:

Hopefully, the thing you mentioned with the Steam controller configs above will help me get the best of both worlds :crossed_fingers:

I think it's a bit unfair to say a lot of your suggestions are dismissed as out-of-scope.

Thanks :purple_heart:

I'm gradually trying to learn what's in-scope for a project that covers as many different tools as this one :eyes:

And for commandline usage, a Bash script could be written to take in a list of AppIDs perhaps from a CSV and name the game that way.

This sounds pretty cool! :+1:

It is a lot simpler to do this manually than it would be to not only implement this feature, but it would be a lot more reliable to do it manually too, as it is just copy and pasting from the Steam Store Page URL or wherever else one gets their AppIDs, it's not exactly hidden information. I think the user should be responsible for naming the game the way they want. It's a lot easier for a human to get the AppID for a game than a Bash script.

That's fair. It's kind of unrealistic to expect Steam to just know what I'm trying to do when I'm adding a game from outside of its database.

Some tinkering is fine--I don't mind that--I'm just trying to avoid unnecessary work, especially if it can be automated. If it can't, though--at least, not reliably--then it's not a big deal to have to copy over one or two things per game, I guess.

sonic2kk commented 1 year ago

The first part of this was implemented with #926, where we now have an option to look in the game EXE directory for artwork based on name. It uses files to try and ensure we're dealing with image files, but doesn't filter by file type or anything, as the file types that Steam supports is not known to me and could expand as time goes on. I know it at least supports png, jp(e)g, and gif, but I'm not sure about others. It may support some AV1 files too.

If there are different file types with the same name (such as logo.png and logo.jpg), STL will use whichever one is returned first by find.

The next part of this is implementing the SteamGridDB call to get artwork. No ETA, just whenever I have time to dig into it.

CartoonFan commented 1 year ago

Much appreciated, good sir :+1:

I know I ask a lot of different things, so I appreciate your work and input, as always :laughing:

CartoonFan commented 1 year ago

The first part of this was implemented with #926, where we now have an option to look in the game EXE directory for artwork based on name. It uses files to try and ensure we're dealing with image files, but doesn't filter by file type or anything, as the file types that Steam supports is not known to me and could expand as time goes on. I know it at least supports png, jp(e)g, and gif, but I'm not sure about others. It may support some AV1 files too.

It's not a comprehensive addition to the list, but APNG and WebP are also supported by SteamGridDB (not sure about Steam, though :shrug:):

https://www.steamgriddb.com/grid/93056 https://www.steamgriddb.com/grid/121948

The next part of this is implementing the SteamGridDB call to get artwork. No ETA, just whenever I have time to dig into it.

Got it 🫡

sonic2kk commented 1 year ago

It's not a comprehensive addition to the list, but APNG and WebP are also supported by SteamGridDB (not sure about Steam, though 🤷)

Oh this is interesting, at least for the ones you linked, APNG is flagged by the file command as containing image data, but the WebP file is not!

$ file nier_apng.png
nier_apng.png: PNG image data, 600 x 900, 8-bit/color RGB, non-interlaced
$ file nier_webp.webp
nier_webp.webp: RIFF (little-endian) data, Web/P image

I guess I'll update the results check to include Web/P image as well as image data once I have time. Thanks for pointing that out!

CartoonFan commented 1 year ago

It's not a comprehensive addition to the list, but APNG and WebP are also supported by SteamGridDB (not sure about Steam, though 🤷)

Oh this is interesting, at least for the ones you linked, APNG is flagged by the file command as containing image data, but the WebP file is not!

$ file nier_apng.png
nier_apng.png: PNG image data, 600 x 900, 8-bit/color RGB, non-interlaced
$ file nier_webp.webp
nier_webp.webp: RIFF (little-endian) data, Web/P image

I guess I'll update the results check to include Web/P image as well as image data once I have time. Thanks for pointing that out!

No problem! I always seem to run into the weird cases somehow :laughing:

CartoonFan commented 1 year ago

Found one more type--ICO files (for icons):

https://www.steamgriddb.com/icon/11551

It also seems like the supported image types are documented in the API:

https://www.steamgriddb.com/api/v2#tag/GRIDS

Hopefully that should be everything this time :crossed_fingers: :grin:

sonic2kk commented 1 year ago

Thanks! .ico files return image data, so the outlier here is WebP it seems.

CartoonFan commented 1 year ago

Thanks! .ico files return image data, so the outlier here is WebP it seems.

Glad to help! 🫡

sonic2kk commented 1 year ago

So some good news on this and a little bit of bad news.

The good news is that 90% of the functionality we need is already implemented with SteamTinkerLaunch's SteamGridDB functionality. A little bit of tweaking allows for using a custom API endpoint with the getGrids function, and a separate function to call for this. Since we can pass it a specific endpoint to hit we can just pass this function the SteamGridDB Game ID and it will download the grid! Locally, I have this implemented. There is logic somewhere in STL to automatically copy the grids over to the Steam grids folder, so I could probably re-use that and/or implement some logic to do this for Non-Steam Games in the new function to call getGrids with the general endpoint.

Right now, STL just fetches the most popular grid, which I think is fine and pretty much what I was planning to do here anyway. We can't give the user a way to pick really from the Non-Steam Game UI or commandline. However, the situation is not as dire as one might think!

Since we can re-use the getGrid function which basically means re-using 90% of what STL already does, that means the Steam Grid settings on the Global Menu will apply here as well! Users can set some filters to get some control over image size, content and so on, since those are all global settings and are used by the function. The configuration the user does here will also apply to Non-Steam Games (except for the copying to the Steam Grids folder bit, for now).

Some mixed news is that SteamTinkerLaunch's SteamGridDB functionality only downloads one grid, and it only downloads boxart (1223123p.png files). We'll need to do a rework of getGrids to allow for this. It looks like for each image type, SteamGridDB has a different endpoint. We may have to extend dimensions for each of these, filling them in with some sensible defaults, and maybe adding some configuration options so users can choose whether they also want to download grids/heroes/logos/icons/etc.

This is good in one sense, because it means expanding the functionality of the SteamGridDB integration, and since we're reusing parts of the code here, this change would benefit all of STL so regular Steam games would benefit.

However this is mixed in the sense that the actual getGrid function would need retooled for it, since it's pretty specifically set to handle only one type of image. We'll need a rework to fetch multiple grids.

Now the bad news: The SteamGridDB API is, well, painfully slow for large grids. I totally understand this from a development perspective and from SteamGridDB's perspective, bandwidth ain't cheap. But for animated grids, like the NieR APNG one which is like 60mb, it took about 25 minutes to download. For "regular" grids so to speak, that is standard PNGs and JPEGs, or even smaller WebP files, they download in a second or two.

One other final note is that SteamGridDB needs an API key to work. This is to prevent spam and probably so they can track which requests are coming and how frequently, it's pretty standard with APIs. But I think sine this feature may be more apparent we should make it a bit clearer that you need to supply your own. It's already noted on the SteamGridDB wiki page, it's also noted where to set it, but we should make it a bit clearer. This can be done from the SteamGridDB Profile Preferences section, under the API tab (it actually took me a while to figure that out). Once generated this can be set in STL either on the Global Menu, manually in the text file, or with steamtinkerlaunch set SGDBAPIKEY global '<apikey>'.

We can't supply one really, because it could be taken and used, and since STL is just a Bash script that can't really be compiled, we'll just have to get users to supply their own. This isn't new, this is how the existing SteamGridDB functionality works, but yeah it's just something I'll have to remember to make clearer. The link STL provides on where to generate the API key is also wrong (outdated?), so I'll fix that too.

The API also probably has a limit, and this is something that should be kept in mind for users calling this function from the commandline, so I'll note on the wiki under the commandline section or the eventual SteamGridDB integration section.

So to summarise, this should be straightforward to implement for Non-Steam Games with the current SteamGridDB support, and will have all the configuration benefits of the existing integration! However, only one image will be downloaded, and if the default option of allowing animated grids is left on, then downloads could take an unreasonable amount of time, but this is not a SteamTinkerLaunch limitation (if you got a magical API token with no throttling you wouldn't have this issue). All of the limitations of the existing SteamGridDB integration will apply to Non-Steam Games, though this has brought it to my attention that there are possible improvements which can be made to how STL integrates with it. And once those improvements are in place, Non-Steam Games will benefit as well :-)

sonic2kk commented 1 year ago

Initial PR is up at #927, which for now will implement the goal of this issue: Add SteamGridDB support for Non-Steam Games. Even though it only adds boxart, the rework to allow other artworks to be fetched from SteamGridDB is a broader rework, which I'll write up a separate issue for this weekend hopefully, with some implementation details. In my tests, this appeared to work.

image

That rework will by default encompass Non-Steam Games, since it will be a general rework of how we fetch grids. The function call we make now won't have to change but what it does will change, but we also don't have to pass it any other information so what we give it now will work once I get around to overhauling the SteamGridDB stuff. That'll be a fun task, there were a couple of TODOs left there 19 months ago that I should be able to knock out as well :-)

There may also be some other changes around SteamGridDB and the artwork section for Add Non-Steam Game and Set Game Artwork. I had the idea since SteamGridDB doesn't give a way to fetch tenfoot artworks, that we could add a checkbox option try to re-use/crop/resize (with imagemagick most likely) some game hero or boxart to be used as a tenfoot as a fallback if none is found or provided in these instances. I'll have to do some tests and see how that would look when done manually for games (both in the Steam Client and in Big Picture, if Big Picture doesn't already re-use the hero artwork by default).

So once the PR is polished up a little more I'll merge it and write up some more about the remaining work in this area (and maybe write up some tickets to detail other ideas I have but may not implement for a while). Shouldn't be that much of a pain hopefully but we'll see :-) Refactoring is always... a Pepsi-intensive activity.

CartoonFan commented 1 year ago

Initial PR is up at #927, which for now will implement the goal of this issue: Add SteamGridDB support for Non-Steam Games. Even though it only adds boxart, the rework to allow other artworks to be fetched from SteamGridDB is a broader rework, which I'll write up a separate issue for this weekend hopefully, with some implementation details. In my tests, this appeared to work.

Sounds like you've got your work cut out for you :grin:

That rework will by default encompass Non-Steam Games, since it will be a general rework of how we fetch grids. The function call we make now won't have to change but what it does will change, but we also don't have to pass it any other information so what we give it now will work once I get around to overhauling the SteamGridDB stuff. That'll be a fun task, there were a couple of TODOs left there 19 months ago that I should be able to knock out as well :-)

Best of luck! :+1:

There may also be some other changes around SteamGridDB and the artwork section for Add Non-Steam Game and Set Game Artwork. I had the idea since SteamGridDB doesn't give a way to fetch tenfoot artworks, that we could add a checkbox option try to re-use/crop/resize (with imagemagick most likely) some game hero or boxart to be used as a tenfoot as a fallback if none is found or provided in these instances. I'll have to do some tests and see how that would look when done manually for games (both in the Steam Client and in Big Picture, if Big Picture doesn't already re-use the hero artwork by default).

Ooh, sounds cool! I'll be here to help test things, if you want.

So once the PR is polished up a little more I'll merge it and write up some more about the remaining work in this area (and maybe write up some tickets to detail other ideas I have but may not implement for a while). Shouldn't be that much of a pain hopefully but we'll see :-) Refactoring is always... a Pepsi-intensive activity.

Don't forget to listen to your body's needs :smile:

It always seems to be one thing after another with these issues, but I guess that's the general path forward to improvement here. Are the other issues usually like this? My seemingly simple bugs and feature requests seem to frequently end up more complex by the time they're finished :thinking: :sweat_smile:

sonic2kk commented 1 year ago

It always seems to be one thing after another with these issues, but I guess that's the general path forward to improvement here

Yup, there may be an element of over-engineering sometimes, but at the end of the day I'm also a user of SteamTinkerLaunch (who ended up becoming maintainer 😄) so I also think from the perspective of what I would find useful.

Are the other issues usually like this? My seemingly simple bugs and feature requests seem to frequently end up more complex by the time they're finished

Well, there are a lot of not great issues opened that don't warrant discussion (blank, no response, entitled attitudes). Sometimes bug reports are just pointing the user in the right direction of something, like clearing some cache files.

But having said that, "good quality" bug reports/feature requests can regularly grow in complexity, or at the very least end up with a lot of discussion. Sometimes the fix is simple but lots of discussion is needed. Sometimes to implement a feature at a basic level it could be straightforward, but to implement it well or with other functionality around it to make it more useful, that can take a bit more work.

For the most recent examples here, SpecialK+ReShade and SteamGridDB, they were less used and so got less attention, and they were both implemented about two years ago and untouched since. So it's less I think that your issues are complex per se and that you're just helping uncover the room for improvement. There always is room for improvement of course but because these are the less used aspects of STL they haven't gotten as much love. But, with these improvements, that could change things!

Outside of these, I opened an issue long ago about Origin games not working and it was a non-trivival investigation task for the previous maintainer (Origin broke again likely as a result of some DRM-specific handoff). Another example more recently would be the Vortex Mod Manager madness that required some very in-depth changes to how Vortex was downloaded and installed. Earlier this year I really wanted to improve One-Time Run and there was a huge refactor there basically rewriting most of the code around it (#788). GameScope has seen three significant refactors. Even more recent again, #905 required a big research effort for how to write out to VDF files in Bash with the specific format required, as well as getting the required names. Predating that issue and this one, #738 was a beast to figure out, it took me a year pretty much!

Hope that clears things up a bit! I personally welcome the issues as a user of STL and a developer who wants to keep trying to improve this project. Things won't get fixed/improved if people don't get involved :-)

CartoonFan commented 1 year ago

It always seems to be one thing after another with these issues, but I guess that's the general path forward to improvement here

Yup, there may be an element of over-engineering sometimes, but at the end of the day I'm also a user of SteamTinkerLaunch (who ended up becoming maintainer 😄) so I also think from the perspective of what I would find useful.

Unfortunately, I usually seem to forget this :pray:

With that perspective, I don't feel as bad asking for stuff :sweat_smile:

Are the other issues usually like this? My seemingly simple bugs and feature requests seem to frequently end up more complex by the time they're finished

Well, there are a lot of not great issues opened that don't warrant discussion (blank, no response, entitled attitudes). Sometimes bug reports are just pointing the user in the right direction of something, like clearing some cache files.

But having said that, "good quality" bug reports/feature requests can regularly grow in complexity, or at the very least end up with a lot of discussion. Sometimes the fix is simple but lots of discussion is needed. Sometimes to implement a feature at a basic level it could be straightforward, but to implement it well or with other functionality around it to make it more useful, that can take a bit more work.

For the most recent examples here, SpecialK+ReShade and SteamGridDB, they were less used and so got less attention, and they were both implemented about two years ago and untouched since. So it's less I think that your issues are complex per se and that you're just helping uncover the room for improvement. There always is room for improvement of course but because these are the less used aspects of STL they haven't gotten as much love. But, with these improvements, that could change things!

I guess that's the silver lining to just...stumbling into issues. It's honestly a bit of a personal trend with me :sweat_smile:

Outside of these, I opened an issue long ago about Origin games not working and it was a non-trivival investigation task for the previous maintainer (Origin broke again likely as a result of some DRM-specific handoff). Another example more recently would be the Vortex Mod Manager madness that required some very in-depth changes to how Vortex was downloaded and installed. Earlier this year I really wanted to improve One-Time Run and there was a huge refactor there basically rewriting most of the code around it (#788). GameScope has seen three significant refactors. Even more recent again, #905 required a big research effort for how to write out to VDF files in Bash with the specific format required, as well as getting the required names. Predating that issue and this one, #738 was a beast to figure out, it took me a year pretty much!

Woah. That's a lot of work :eyes:

Hope that clears things up a bit! I personally welcome the issues as a user of STL and a developer who wants to keep trying to improve this project. Things won't get fixed/improved if people don't get involved :-)

Thanks :relaxed:

I mean, if it's helping in the long run, I guess it's not a bad thing. Besides this and the STL bits from #913--whatever they might be--I can't really think of much else to suggest right now. STL's pretty feature-packed, and most of the issues so far have been with the specific tools it connects to, rather than with STL itself.

There was one thing I was curious about, however:

Earlier, you briefly mentioned:

This feature is just entering the game AppID as the game name, I don't see why this needs a dedicated option which is why I don't think it needs to be added. If someone wants it badly enough in the future they could open a PR, but it's not something I think is so crucial it needs its own checkbox or flag to automate. At that point, we would open the door to all sorts of options, like taking in a Steam AppID and setting the game name based on that. It becomes a bit much in my mind to try and account for all of these use-cases in a tinker utility, where these options can be easily done with a copypaste. And for commandline usage, a Bash script could be written to take in a list of AppIDs perhaps from a CSV and name the game that way.

This Bash script idea--would that be something I could build on my end? I've had a bit of experience with both Bash scripting and CSVs, but I don't really have an idea of how to accomplish that. I've only made small, simple scripts before--certainly nothing as complex as STL :sweat_smile:

Thanks! :purple_heart: :pray:

sonic2kk commented 1 year ago

PR #927 was merged, so for now this issue is resolved until SteamGridDB integration is extended to include other grids :smile: Once I have some time I will write up an issue or just open a PR to implement this.

This Bash script idea--would that be something I could build on my end?

Yeah, it could for sure. It depends on what way you want to accomplish it. If you have a bunch of games you could create/generate a list of CSVs using Bash, and then essentially map the game name/exe/etc to the Steam AppID. The CSV might look something like this:

steam_id,exe,startdir,compatibility_tool,steamgrid_game_id  # this line is for clarity only, don't include in CSV
524220,/path/to/game.exe,/path/to/startdir,GE-Proton8-16,13971
638970,/path/to/game.exe,/path/to/startdir,GE-Proton8-16,18449
# ... etc

You would then read this CSV and inside of a loop, parse out each section of the CSV and call steamtinkerlaunch ansg. Something like:

# Call steamtinkerlaunch command to create Non-Steam Game using data from each line in the CSV file
# Will name the Non-Steam Games after the AppIDs in the first column, which will accomplish what you want
# It doesn't have to be as fancy as this, or it could be fancier with some automated script to create this CSV from a list of game names
# This is also entirely untested, just a top-of-my-head example of what could be done, to illustrate how you might store the data in a CSV and how you might use this data in a STL commandline call
while read -r csvline; do
    csv_appname="$( echo "$csvline" | cut -d ',' -f1 )"
    csv_exe="$( echo "$csvline" | cut -d ',' -f2 )"
    csv_startdir="$( echo "$csvline" | cut -d ',' -f3 )"
    csv_compattool="$( echo "$csvline" | cut -d ',' -f4 )"
    csv_steamgrid_id="$( echo "$csvline" | cut -d ',' -f5 )"

    # More options than this are available, see "steamtinkerlaunch help" for a full list
    steamtinkerlaunch ansg --appname="$csv_appname" --exepath="$csv_exe" --startdir="$csv_startdir" --compatibilitytool="$csv_compattool" --steamgriddb-game-id="$csv_steamgrid_id"
done <<< "$CSVFILE"

You could get fancier and use some Steam API calls to generate this file from a list of game names (or STL for owned games with steamtinkerlaunch gi "game name"). If they're owned games you could have a file with a list of game names, then do something like this:

# Text file with game names -- The more exact the name, the better, including any unicode characters and punctuation
NieR:Automata
Yakuza 0
The Elder Scrolls IV: Oblivion
Sonic Adventure
Uriel's Chasm
while read -r gameline; do
    # Very simple example, does not handle if ID is not found -- Some regex check to see if it's an AppID may work before writing out
    game_id="$( steamtinkerlaunch gi "$gameline" | head -n1 | cut -d '(' -f1 |  xargs )"

    # ... You could also generate the Game EXE location and so on here too, or simply write them out
    # Instead of echo you'd use printf, something like:
    # printf "%s,%s,%s,%s,%s" "$game_id" "$game_exe" "$game_startdir" "$game_compattool" "$game_steamgriddb_game_id" >> "gamecsv.txt"

    echo "$game_id" >> "gamecsv.txt"
done <<< "$gamenames.txt"

You could do this outside of Bash, too, but since STL is a Bash utility it may be easier to integrate into another script rather than making system calls from another language.

If you're used to working with text files in Bash then hopefully this gives you a clearer idea of how you might actually do this in combination with STL. And of course this is not the only or best way, just a small demonstration.

STL has a lot of commandline functionality, at its core it is a commandline tool. Most of the internal calls are exposed as commandline options where it makes sense, and much of the UI, like the Add Non-Steam Game GUI, is just a frontend for the command. When you click the "Create" button, STL actually makes a call to the commandline the exact same way that script would, just with the variables representing the values entered on the UI.

# Called at the end of addNonSteamGameGui
addNonSteamGame -an="$NOSTGAPPNAME" -ep="$NOSTGEXEPATH" -sd="$NOSTGSTDIR" -ip="$NOSTGICONPATH" -lo="$NOSTGLAOP" -hd="$NOSTGHIDE" -adc="$NOSTGADC" -ao="$NOSTGAO" -vr="$NOSTGVR" -t="$NOSTTAGS" -ct="$NOSTCOMPATTOOL" -hr="$NOSTGHERO" -lg="$NOSTGLOGO" -ba="$NOSTGBOXART" -tf="$NOSTGTENFOOT" "--${NOSTGSETACTION}" "$NOSTARTEXECMD" -sgid="$NOSTSGDBAID"

Thanks! And please don't feel bad about opening issues, I appreciate them :-)

CartoonFan commented 1 year ago

Very nice. I feel like I learned something here :grin:

I'll have to go through this more slowly when I'm trying it out for myself, but having it here like this is super helpful.

If you're used to working with text files in Bash then hopefully this gives you a clearer idea of how you might actually do this in combination with STL.

I'm not, but I think I get the gist of what's here :sweat_smile:

When you click the "Create" button, STL actually makes a call to the commandline the exact same way that script would, just with the variables representing the values entered on the UI.

# Called at the end of addNonSteamGameGui
addNonSteamGame -an="$NOSTGAPPNAME" -ep="$NOSTGEXEPATH" -sd="$NOSTGSTDIR" -ip="$NOSTGICONPATH" -lo="$NOSTGLAOP" -hd="$NOSTGHIDE" -adc="$NOSTGADC" -ao="$NOSTGAO" -vr="$NOSTGVR" -t="$NOSTTAGS" -ct="$NOSTCOMPATTOOL" -hr="$NOSTGHERO" -lg="$NOSTGLOGO" -ba="$NOSTGBOXART" -tf="$NOSTGTENFOOT" "--${NOSTGSETACTION}" "$NOSTARTEXECMD" -sgid="$NOSTSGDBAID"

That...is a pretty long call :sweat_smile:

Thanks! And please don't feel bad about opening issues, I appreciate them :-)

I hope I'll remember this the next time it happens :laughing:

But seriously: thank you.

sonic2kk commented 1 year ago

Okay, a bit delayed but I finally got around to thinking out and writing up a proposal that should encompass all of the ideas I have for improving the SteamGridDB integration (#933). I hope maybe that issue can help make a bit more sense of why I closed this one, as the scope of changes has far outgrown simply just integrating SteamGridDB for Non-Steam Games.

And because of the way the code is shared, it doesn't really make sense to implement for Non-Steam Games but not for Steam Games. It could be done, but a refactor makes more sense, because then it's easier to maintain for one but also there are more benefits. If we can download and set SteamGridDB logo+hero+boxart for Steam games by overhauling the existing functionality, then it's just a case of passing the SteamGridDB Game ID for Non-Steam Games, and the code will handle that automagically.

There are other improvements that can be made too on this, such as automatically using/resizing a tenfoot image (the 600x350 image used in regular Steam mode for the most recently played game). I also want to just generally improve how SteamGridDB is used throughout SteamTinkerLaunch as a result of this, because it could be very nice to have. Being able to more easily use STL to set artwork for one specific game would be heaven :smile:

But seriously: thank you.

No problem :-) Sometimes changes take a while to think through, discuss, test and ultimately implement, but it's all good fun at the end of the day. And as long as it's fun and I keep using STL, I will keep trying to improve it!

CartoonFan commented 1 year ago

And because of the way the code is shared, it doesn't really make sense to implement for Non-Steam Games but not for Steam Games. It could be done, but a refactor makes more sense, because then it's easier to maintain for one but also there are more benefits. If we can download and set SteamGridDB logo+hero+boxart for Steam games by overhauling the existing functionality, then it's just a case of passing the SteamGridDB Game ID for Non-Steam Games, and the code will handle that automagically.

Totally understand. I support this approach :+1:

No problem :-) Sometimes changes take a while to think through, discuss, test and ultimately implement, but it's all good fun at the end of the day.

It's usually worth the wait, even if I get impatient sometimes :rofl:

And as long as it's fun and I keep using STL, I will keep trying to improve it!

A noble goal! :muscle:

Thanks again! :purple_heart: :laughing:

sonic2kk commented 1 year ago

Hey @CartoonFan, after quite the wait, the initial overhaul of SteamGridDB support is now in SteamTinkerLaunch, with v14.0.20231018-1! The main change with regards to this issue is that all artwork is now downloaded when the SteamGridDB Game ID is given! It will also download and set the icon for games. I have tested this myself with a few games and combined with the option to select the compatibility tool, I have to say it is pretty sweet to have nicely decorated Non-Steam Games added to the client with a compatibility tool set out of the box, so the entries in the library look pretty and are usable right away.

In terms of Non-Steam Games, there is no changes required from before, it should work right away. There is also a notifier to tell you when the download from SteamGridDB begins and ends. The artwork settings can all be configured on the Global Menu, though artwork will always be applied automatically for Non-Steam Games. SteamGridDB will only be searched when the SteamGridDB Game ID is passed, and any artwork found will always be downloaded into the Steam Grid folder, even if this option is left at the default disabled for all other games on the Global Menu.

You can also set artwork for Steam games or even Non-Steam Games provided you have their AppID via the commandline, and you aren't limited to only searching by AppID. You can search using the SteamGridDB Game ID for one game, but tell the command to apply it to a game with another AppID. One nice example of this is getting grid artwork for "Sonic Colors" from SteamGridDB, using the SteamGridDB Game ID for the Wii game (or even "Sonic Colours"), instead of getting artwork for "Sonic Colors: Ultimate". The main benefits here are that you could get "Sonic Colours", with the "u", but also that this entry has much more artwork than "Sonic Colors: Ultimate", so you can get more flexibility.

The SteamGridDB wiki page got a massive overhaul documenting how to use the command.

Mostly we use the defaults from SteamGridDB's searches i.e. keeping the same dimensions, but we make a couple of changes. For tenfoot artwork, which appears as the banner artwork for the most recently played game, I chose to exclude any tenfoot artwork that was tagged with no_logo. This is because if there is no logo, it may not be obvious which game the artwork is for, so as a aesthetic choice I chose to enforce only tenfoot artwork that has a logo on it. I mention this because there could be a rare case where no artwork of this criteria is met. In this case you can manually add no_logo to the list of styles for tenfoot artwork. I tested about 30 games and didn't come across any like this, but I wanted to make note of it (it's also noted on the wiki).

Any artwork can also be manually overwritten from the Steam Desktop client if you're not happy with whatever is chosen as well, sometimes the least popular artwork on the very last page on SteamGridDB happens to be my favourite :-) But if you aren't picky or just want something to begin with instead of grey library entries, it's much better than nothing (and for games with minimal artwork, like Touhou Fumo Racing, it works great for my tastes :smile:).

In terms of future improvements, specifically relating to Non-Steam games I have two in mind:

Hopefully that gives you an update on where we are right now overall, and what future improvements specifically relating to Non-Steam Games are planned for the future. Hopefully it's useful to you in its current form and continues to get more useful :-) You can track the progress over at #933.

CartoonFan commented 1 year ago

Sorry for the delay on my end, as well. There have been quite a number of small-scale issues vying for my attention, including more stuff with #894, and a possible bug with running Non-Steam Games that I'll explain more about below :sweat_smile:

I tested the new SteamGridDB stuff with Tokyo Xanadu eX+ (that's the actual name, weird capitalization and all), and I was pleasantly surprised!

2023_10_17_16_06_05 2023_10_17_16_05_56

Only thing I did with the SGDB side so far is switch the logo for one with better contrast, but even that was just a few clicks worth of effort! :grin:

If this is what's coming in the future, I'm all for it :+1:

As always, I like to clear issues with you before making new ones, to avoid unnecessary bug reports. After all, I could just be missing something obvious :sweat_smile:

While testing this game, I noticed something strange. When the game's start directory is empty, the actual game seemingly can't find the necessary files, despite them being present in the directory:

2023_10_17_18_10_24

2023_10_17_18_06_45

2023_10_17_18_07_23

[jeremiah@arcadia Tokyo Xanadu eX+]$ pwd
/home/jeremiah/games/managed_games/gog/tokyo-xanadu-ex/drive_c/GOG Games/Tokyo Xanadu eX+
[jeremiah@arcadia Tokyo Xanadu eX+]$ ls -lv
total 5073892
-rw-r--r-- 1 jeremiah jeremiah  877325813 Oct 14 23:55  Asset1.bra
-rw-r--r-- 1 jeremiah jeremiah  499000182 Oct 14 23:54  Asset2.bra
-rw-r--r-- 1 jeremiah jeremiah 1653278165 Oct 14 23:56  Asset3.bra
-rw-r--r-- 1 jeremiah jeremiah  785690772 Oct 14 23:54  Asset4.bra
-rw-r--r-- 1 jeremiah jeremiah  716768383 Oct 14 23:55  Audio.bra
-rw-r--r-- 1 jeremiah jeremiah      51867 Oct 17 10:32  EULA.txt
-rw-r--r-- 1 jeremiah jeremiah    4730880 Oct 14 23:55  Galaxy.dll
-rw-r--r-- 1 jeremiah jeremiah  555182718 Oct 14 23:55  Japanese.bra
-rw-r--r-- 1 jeremiah jeremiah        633 Oct 14 23:58 'Launch Tokyo Xanadu eX+.lnk'
-rw-r--r-- 1 jeremiah jeremiah        277 Oct 15 00:15  ReShade.ini
-rw-r--r-- 1 jeremiah jeremiah     189241 Oct 17 17:25  ReShade.log
-rw-r--r-- 1 jeremiah jeremiah    3222240 Oct 17 16:03  ReShade32.dll
-rw-r--r-- 1 jeremiah jeremiah        526 Oct 17 18:06  ReShade32.json
drwxr-xr-x 3 jeremiah jeremiah       4096 Oct 17 17:11  Screenshots
-rw-r--r-- 1 jeremiah jeremiah         99 Oct 17 18:06  SpecialK_enabled.txt
-rw-r--r-- 1 jeremiah jeremiah   69753379 Oct 14 23:55  System.bra
-rwxr-xr-x 1 jeremiah jeremiah    7225344 Oct 14 23:55  TokyoXanadu.exe
-rw-r--r-- 1 jeremiah jeremiah       1755 Oct 17 17:13  TokyoXanadu.ini
-rw-r--r-- 1 jeremiah jeremiah    1574072 Oct 17 17:25 'TokyoXanadu 2023-10-17 17-25-05.png'
-rw-r--r-- 1 jeremiah jeremiah     121589 Oct 17 10:21  TokyoXanadu_20231017_172150.dmp
-rw-r--r-- 1 jeremiah jeremiah     119695 Oct 17 10:22  TokyoXanadu_20231017_172247.dmp
-rw-r--r-- 1 jeremiah jeremiah      46921 Oct 17 17:07  TokyoXanadu_20231018_000746.dmp
-rw-r--r-- 1 jeremiah jeremiah         12 Oct 15 00:15  check-steam_appid.txt
-rw-r--r-- 1 jeremiah jeremiah    3681600 Oct 17 16:03  d3dcompiler_47.dll
-rw-r--r-- 1 jeremiah jeremiah   10544128 Oct 17 18:06  dxgi.dll
-rw-r--r-- 1 jeremiah jeremiah       7197 Oct 17 18:06  dxgi.ini
-rw-r--r-- 1 jeremiah jeremiah      69248 Aug  9  2017  gog.ico
-rw-r--r-- 1 jeremiah jeremiah        157 Oct 17 10:32  goggame-1565811574.hashdb
-rw-r--r-- 1 jeremiah jeremiah     179429 Mar 30  2018  goggame-1565811574.ico
-rw-r--r-- 1 jeremiah jeremiah        336 Oct 17 10:32  goggame-1565811574.info
-rw-r--r-- 1 jeremiah jeremiah        157 Oct 14 23:56  goggame-1908505665.hashdb
-rw-r--r-- 1 jeremiah jeremiah     179429 Mar 30  2018  goggame-1908505665.ico
-rw-r--r-- 1 jeremiah jeremiah        603 Oct 14 23:56  goggame-1908505665.info
drwxr-xr-x 3 jeremiah jeremiah       4096 Oct 17 18:06  logs
drwxr-xr-x 2 jeremiah jeremiah       4096 Oct 14 23:55  movie
drwxr-xr-x 2 jeremiah jeremiah       4096 Oct 14 23:56  movieJP
drwxr-xr-x 5 jeremiah jeremiah       4096 Oct 15 00:14  reshade-shaders
-rw-r--r-- 1 jeremiah jeremiah         11 Oct 15 00:15  steam_appid.txt
-rw-r--r-- 1 jeremiah jeremiah      62895 Aug  9  2017  support.ico
-rw-r--r-- 1 jeremiah jeremiah    1862360 Oct 14 23:58  unins000.dat
-rwxr-xr-x 1 jeremiah jeremiah    1334880 Oct 14 23:58  unins000.exe
-rw-r--r-- 1 jeremiah jeremiah         41 Oct 14 23:58  unins000.ini
-rw-r--r-- 1 jeremiah jeremiah      23077 Oct 14 23:58  unins000.msg
-rw-r--r-- 1 jeremiah jeremiah    1634786 Oct 17 10:32  unins001.dat
-rwxr-xr-x 1 jeremiah jeremiah    1334880 Oct 17 10:32  unins001.exe
-rw-r--r-- 1 jeremiah jeremiah         41 Oct 17 10:32  unins001.ini
-rw-r--r-- 1 jeremiah jeremiah      23077 Oct 17 10:32  unins001.msg
-rw-r--r-- 1 jeremiah jeremiah     298538 Mar 30  2018  webcache.zip

Giving the game dir to Steam kills the game process before Steam Tinker Launch even gets a chance to load:

2023_10_17_18_13_27

Launching the game through One time run produces a similar result to running the game through regular Wine:

2023_10_17_18_17_43

And changing the working directory there does...nothing? No game, and the function doesn't even save the setting:

2023_10_17_18_23_24

The only way I was able to make it work was by running the game as a custom command:

2023_10_17_18_26_22

2023_10_17_18_26_49

2023_10_17_18_26_57

2023_10_17_18_27_24

2023_10_17_18_27_31

Running the game with a Steam compatibility tool (Proton-GE, Proton-TkG, regular Proton, etc.), doesn't help the game find the files, either :shrug:

I'm pretty sure this is a bug of some sort, but, as mentioned, I'm looking for a second opinion :pray:

Logs and stuff

3678379145.log

steam-15798518130096996352.log

3678379145.log

sonic2kk commented 1 year ago

Glad the SteamGridDB stuff is working!

As for the Non-Steam game stuff, I have some thoughts (cant reproduce in light testing but will test a lot more thoroughly in a bit), but it is potentially a bug, so I am going to try and open a separate issue from the comment you left, a feature GItHub offers that I don't believe I've used before. Once opened, please edit it in any way you think would suit, and after that, I'll comment on it. Converting it straight into an issue may not be "ideal" so fill in any gaps/other information you think is missing :-) That was a massive fail, that did not go as expected.

Could you copypaste everything related to the Non-Steam Game into a new issue? Make any edits you feel are necessary but a straight copypaste is fine as well :-) That's what I tried to do, excluding the SteamGridDB stuff.

One extra bit of info that maybe I just missed from your comment that I would like you to include, is what the behaviour is without SteamTinkerLaunch.

CartoonFan commented 1 year ago

Got it! Thanks as always for your hard work :grin:

Also,

sometimes the least popular artwork on the very last page on SteamGridDB happens to be my favourite :-)

I do this too :rofl:

sonic2kk commented 1 year ago

I'm looking into the problem as we speak to be clear, I hope this didn't come off as brushing the problem off. But I didn't want to clutter up this too much because you're probably on to a valid issue. At the very least I think if STL isn't setting the start dir properly then it should and this is a bug (iirc it's supposed to set it to the directory of the exe if no other start directory is selected, because this is what Steam does, and I just confirmed it). But in terms of reproducing the probblem it may go deeper than that and I'll have to do some more investigation, so a separate issue to list out all my investigation is just cleaner :-)

CartoonFan commented 1 year ago

I'm looking into the problem as we speak to be clear, I hope this didn't come off as brushing the problem off. But I didn't want to clutter up this too much because you're probably on to a valid issue. At the very least I think if STL isn't setting the start dir properly then it should and this is a bug (iirc it's supposed to set it to the directory of the exe if no other start directory is selected, because this is what Steam does, and I just confirmed it). But in terms of reproducing the probblem it may go deeper than that and I'll have to do some more investigation, so a separate issue to list out all my investigation is just cleaner :-)

I didn't take it that way, so you're good.

I'm in the middle of setting up the new issue right now :+1: