Reshade always installs into custom command directory when using ONLY_CUSTOMCMD with no way to override

System Information

SteamTinkerLaunch version: steamtinkerlaunch-v14.0.20240420-1
Distribution: Garuda Linux
Installation Method: AUR

Issue Description

this is the same exact issue as https://github.com/sonic2kk/steamtinkerlaunch/issues/669 except its when the user sets ONLY_CUSTOMCMD and wants reshade installed into the game's path

in this case reshade will always be installed into the custom commands directory regardless of CUSTOMCMDRESHADE

When using a custom command INSTEAD of the actual game exe stl installs all files into the path of the custom command.

as you can see here the reshade destination directory is set to be the same as the shader destination directory

SHADDESTDIR gets set to path to the executable which is getting launched which in this case is our custom command.

We actually have a workaround for this, though if we have an ALTEXEPATH then that gets set to SHADDESTDIR (strangely in two places dup?) https://github.com/sonic2kk/steamtinkerlaunch/blob/04ae6e9ea8733d980fe5d78fa93327230a9c34e2/steamtinkerlaunch#L5314

https://github.com/sonic2kk/steamtinkerlaunch/blob/master/steamtinkerlaunch#L9208

The strange thing is, as you can see in my log, I am setting ALTEXEPATH to be the game directory

setShaderDest - Overriding SHADDESTDIR to '/home/zany130/.local/share/Steam/steamapps/common/P3R/P3R/Binaries/Win64' because ALTEXEPATH is set to '/home/zany130/.local/share/Steam/steamapps/common/P3R/P3R/Binaries/Win64'

It even installing the shaders into the directory (without this option set it will also install the shaders it the directory of my custom command)

my guess is that SHADDESTDIR is getting set to ALTEXEPATH it installs the shaders and then somehow SHADDESTDIR gets overwritten again to the custom commands path?

Logs

steamtinkerlaunch.log

P.S

Sorry for the long issue lol I was trying to debug it myself since I still mostly remember how all the reshade stuff worked, but got stuck

Sorry, I'm still not totally clear on what the problem is. As per the v12.0 changelog:

Fix ReShade installing into custom command by default by @sonic2kk - Now ReShade will only install into a custom command if a specific option for it is enabled, or if ReShade is enabled and "Only custom command" is enabled

This was implemented in #696.

So if you enable ReShade and are using a custom command you have to enable this specifically. Setting the alternative EXE path is not specific to using custom commands, it's in the case of a game having a launcher but you want ReShade to apply to the game EXE.

There is an option under ReShade settings to install ReShade for custom commands, maybe this is what you're looking for? From checking the log, it doesn't look like CUSTOMCMDRESHADE appears anywhere in it, so I don't think it's enabled.

I'm still not clear on what the problem is, so sorry if I misunderstood...

Completly forgot about that issue :sweat: but in that issue cheat engine was being forked and then the game runs normally.

Here, what I am doing (and I forgot to mention this in the op) is launching reloaded instead of Persona 3. Reloaded then is loading Persona 3 with mods. I, however, want Reshade to install into Persona 3 directory.

The problem is that reshade (and shaders by default) are being installed into the same directory as reloaded (I guess because that becomes the ABSGAMEEXEPATHDIR) . toggling the use reshade with custom command option seems to have no effect here as it always installs there. ( my guess is because my understanding is that toggle controls weather to install into ABSGAMEEXEPATHDIR or the custom command path?)

I then had the idea of setting the ALTEXEPATH to my game executable and that fixes the shaders being installed Reloaded directory but the actual .dll are still being installed there (which is what I am confused on as .dll directory = shader directory)

Ah, I think I have a handle on the problem now. Thank you for clarifying!

The problem is that the ReShade DLLs aren't being installed to the alternative EXE path, they're always going into the custom command folder?

What happens if you disable "use ReShade with custom command"? I assume the ONLY_CUSTCMD option will kick in and cause the same problem...

My guess here is that alternative EXE paths are ignored for custom commands, and STL is always using the custom command folder to install DLLs for custom commands. So no matter what path you choose, if you use ReShade with custom commands, the custom command folder will always be used, effectively meaning you can't specify a custom EXE path for custom commands.

Shaders get put in the right place but maybe there's an override somewhere as you said that messes this up for custom commands.

Let me know if that sounds right. If so I'll do some testing when I'm home and see if I can find a fix based on the parts of the code you linked (thanks for that btw! 😄)

Yeah pretty much, however to clarify

I am using the ALTEXEPATH as a workaround for the issue (sorry prob caused some confusion there) Like you said above that seems to be more for when you have a launcher in say /home/zany130/.steam/steam/steamapps/common/P3R/ but the actual game exe is in /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ you would want the latter as your path for reshade.

That is not an issue for Persona 3 Reload. Assuming you don't sent a custom command like I have stl will install reshade into /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ which is what I want.

So most likely this is the same exact issue as https://github.com/sonic2kk/steamtinkerlaunch/issues/669 except its when the user sets ONLY_CUSTOMCMD and wants reshade installed into the game's path

I don't believe I ever tested using a custom command instead of the game exe, which then launches the game exe.

I thought if someone was overriding their game exe then new exe would be what they would want reshade on

and I think that's what the code is currently doing.

I was HOPING ALTEXEPATH would force the path to be /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ no matter what.

if that's possible that may be an easier way to handle all the custom command edge cases.

TLDR

this is the same exact issue as https://github.com/sonic2kk/steamtinkerlaunch/issues/669 except its when the user sets ONLY_CUSTOMCMD and wants reshade installed into the game's path

in this case reshade will always be installed into the custom commands directory regardless of CUSTOMCMDRESHADE

EDIT: sorry I was in the mindset of fixing the issue instead of describing the issue

classic X.Y problem

Hmm, this is tricky haha, so I'll try outline my understanding of how things are working. I'm very sorry because to someone that uses ReShade these are probably really obvious, I'm doing my best to get a grasp on the problems.

Here's my understanding of the ReShade Scenarios.

Regular Game
- ReShade enabled
- ReShade installed into either the Game EXE folder, or ALTEXEPATH.
Custom command
- ReShade enabled
- Custom command enabled
- ReShade is installed to the Game EXE folder by default unless:
  - Either, "Use ReShade with Custom Command" is enabled,
  - Or "Only custom command" is enabled
  - If either of these are set, ReShade will be installed into the Custom Command EXE folder, otherwise it will be installed into the game EXE folder as with "Regular Game"
Using ReShade with Custom Command:
- ReShade enabled
- Custom command enabled
- ReShade with Custom Command enabled
- Only Custom Command enabled
- This installs ReShade into the same folder as the Custom Command EXE
- This however means if the Custom Command is a launcher, ReShade is essentially be "installed for the launcher" and it is not being "installed for the game"
- To get ReShade to install for a known EXE that a custom command might launch (such as to get ReShade to install for the Persona 3 Reload game and not for the mod manager), Alt EXE path can be used to specify where to install ReShade.
  - This scenario is pretty similar to if a game's default EXE was a launcher in one directory, but the game EXE was in another, which is what Alt EXE path can be used for.
Custom command with Alt EXE Path
- ReShade enabled
- Custom command enabled
- ReShade with Custom Command enabled
- Only Custom Command enabled
- Alt EXE Path is set to /tmp/tmp_path (an obscure path for illustration purposes)
- This installs ReShade and shaders into Alt EXE Path, which is /tmp/tmp_path
- No files are installed into the Game EXE folder

If I have missed a scenario, or misunderstood one of the scenarios, please point out where the problem is and what the current functionality is lacking/doing incorrectly.

I am using the ALTEXEPATH as a workaround for the issue

I am probably just misunderstanding but it sounds like ALTEXEPATH is working as intended, if it allows you to launch Reloaded II but use ReShade with Persona 3 Reload? ALTEXEPATH is basically there to say: "The EXE that STL is going to launch is not the folder where ReShade should be installed, instead, put it in this folder". To put it another way:

Like you said above that seems to be more for when you have a launcher in say /home/zany130/.steam/steam/steamapps/common/P3R/ but the actual game exe is in /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ you would want the latter as your path for reshade.

I'm going to play Devil's advocate here, but if we take a step back and view Reloaded II as a launcher, we can reword your above as this:

Like you said above that seems to be more for when you have a launcher in say /home/zany130/mods/Persona 3 Reload/Release but the actual game exe is in /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ you would want the latter as your path for reshade.

So to bring it back to the tl;dr of your reply:

this is the same exact issue as https://github.com/sonic2kk/steamtinkerlaunch/issues/669 except its when the user sets ONLY_CUSTOMCMD and wants reshade installed into the game's path

It sounds to me like using ALTEXEPATH is the intended solution here.

Do you think ONLY_CUSTOMCMD should not influence where ReShade is installed? I can see it either way.

If a user has enabled ReShade and selected to only use the custom command, it may be a bit unusual for it to install into the game folder and they may not know why ReShade is not being used out of the box for their custom commands. It's different when a custom command is launched with a game (default behaviour) rather than in place of it (ONLY_CUSTOMCMD=1), that's why the "ReShade with Custom Command" checkbox exists: To tell STL to use ReShade with the custom command and not just the game. But with ONLY_CUSTOMCMD I figured if ReShade was on, we should treat the Custom Command EXE as much like a replacement for the game EXE as possible, which is likely why ReShade is always installed to the Custom Command EXE folder when ONLY_CUSTOMCMD is enabled.

Conversely, if they were playing a game with ReShade and then they later switch to using a custom command and only that custom command, ReShade would get installed for that custom command by default, without "Use ReShade with Custom Command" being enabled.

But the behaviour for this is noted on the "Use ReShade with Custom Command" checkbox, which states this option is ignored if Only Custom Command is selected.

Yet, this behaviour is not noted on the ReShade wiki page's Custom Command section.

I really have no strong feeling either way, you would probably know a lot more about how a user would expect ReShade and Custom Commands to interact. In https://github.com/sonic2kk/steamtinkerlaunch/issues/669#issuecomment-1321277689 I outlined this behaviour, so the way things are implemented now are "by design" in that we intentionally install ReShade for the Custom Command if Only Custom Command is selected, but this becomes problematic if the Custom Command essentially acts as a launcher (which could happen with other mod launchers used as custom commands, for example, SA2ModManager).

It overall sounds like you would not expect ONLY_CUSTOMCMD to cause ReShade to be installed into the Custom Command folder, which is of course totally fair, but that also means if I'm understanding correctly that nothing is "broken" so-to-speak. I'm not trying to argue semantics over whether or not this is a bug but rather trying to get a handle on whether something is not working because there are quite a few different constellations of using ReShade, and I'm trying to get an understanding on whether or not one of them is broken or if it's that they should be doing something else for more intuitive behaviour.

If all of that is correct, then I guess we just need to discuss what behaviour is better: The existing behaviour, or if we should stop ONLY_CUSTOMCMD from influencing ReShade installation location. And again, I won't really have a problem with any approach, but I would say that maybe to me I am very slightly leaning towards the existing "workaround". And the reason I'm leaning that way is because I view this very similar to if a game was using a launcher and you wanted ReShade installed into the Game EXE instead of the Launcher EXE, because in this scenario the Custom Command can basically be seen as a launcher in my mind.

But my word is not final on this, I'm very open to discussion!

yup, it seems like we are pretty much on the same page now.

when Reshade got install to Reloaded directory I thought "oh since Im using only custom command the assumption is I want reshade on the custom command. Let me use ALTEXEPATH to override that" Which doesn't work for the DLLs

But thinking more about it, I think we should never assume the user wants reshade on the custom command unless they specifically enable the toogle for it. My reasoning is that I can't really think of a scenario where you would really want the custom command path to be the reshade path.

The following is my opinion, the world of PC gaming (and moding) is vast and I may be completely and uterly wrong about this

Just to clarify (it sounds like we are already on the same page on this though)

you never want resahde to install to the same path as a launcher, always to the actually executable you will be running and moding (99% of the time this is the game exe)

Most custom commands user's would run are some sort of mod manager or trainer. that is to say the custom command will never be a game

I guess in theory you could have say a patched game exe (I think old games where moded like this? you would patch the actually executable?) that you want living next to your normal exe so for example /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/P3R_MODED.exe but even then the "MODED" game exe will most likely live in the same directory as the original.

I feel bad after you put all the work into reshade with custom command (and Im not saying to ditch that we can leave it)

but IMO it makes more sense to always install to the game directory and whenever you want to change where reshade is installed (be it because its installing at the wrong sub directory because of a launcher, or because you want it to use your custom command path for some reason) use ALTEXEPATH

the problem is ALTEXEPATH is not being used (or is maybe being overwritten) for the actual dll installation (something I should test is if this is only happening with custom commands or if this is happening also in the scenario where reshade gets installed to the launcher exe not the game)

TLDR

so to answer

I'm not trying to argue semantics over whether or not this is a bug but rather trying to get a handle on whether something is not working because there are quite a few different constellations of using ReShade, and I'm trying to get an understanding on whether or not one of them is broken or if it's that they should be doing something else for more intuitive behaviour.

its both.

Ignoring whether it makes sense to install into the custom command directory.

I haven't been able to get stl to install reshade DLLs into /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ it is instead going into /home/zany130/mods/Persona 3 Reload/Release/

and I have found no way to override this through stl. ALTEXEPATH only overrides the destination for the shaders folder not the actual dll (which is a bug I think I need to test whether the scenario of stl installing into a games launcher directory instead of the final exe is also broken)

I also personally think we should never assume anything reshade (or special (actually have no idea how this plays with all the custom command stuff last I tried specialk doesn't work with P3R plus when you use specialk + reshade now you lose access to the depth buffer) or anything similar) is expected to go anywhere other than the original game executable

but before anything else I think ALTEXEPATH not overriding the path for the DLLs should be looked at

This issue probbbly should of been two separate issues :sweat_smile:

1.[BUG] ALTEXEPATH is not overriding where ReShade DLLs get installed when using ONLY a custom command.

[UX] I think its bad UX to install ReShade DLLs into the custom command directory when only custom command is enabled

Ahhh I gotcha now, okay, now we are definitely on the same page!

I haven't been able to get stl to install reshade DLLs into /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/ it is instead going into /home/zany130/mods/Persona 3 Reload/Release/

So ReShade DLLs when used with custom commands will not obey ALTEXEPATH. That's where the links you posted come into play. There is a bug here where ALTEXEPATH is not used for ReShade DLL installation, and it is possible it gets overwritten.

This definitely needs looked at before UX changes. Putting those changes aside, let's make sure I'm understanding properly:

Observed: ReShade DLLs are always put into the folder of the custom command
Expected: ReShade DLLs should be put into the folder defined by ALTEXEPATH, which is the behaviour for

actually have no idea how this plays with all the custom command stuff last I tried specialk doesn't work with P3R

If it doesn't work in general, then I guess that's an upstream problem. Maybe the selected DLL is wrong and you need to change the backend? P3R is unfortunately DX12 I believe so you may have to choose a different SpecialK DLL name.

However, it would be best to confirm if SpecialK works with P3R when not launched via a mod manager, as Special K is very funny about loading when a launcher is used even on Windows (SpecialK won't load with STEINS;GATE which uses a launcher because it tries to load into the wrong EXE).

That one should be a separate issue though :sweat_smile:

I'll quote a couple messages here:

Most custom commands user's would run are some sort of mod manager or trainer. that is to say the custom command will never be a game I guess in theory you could have say a patched game exe (I think old games where moded like this? you would patch the actually executable?) that you want living next to your normal exe so for example /home/zany130/.steam/steam/steamapps/common/P3R/P3R/Binaries/Win64/P3R_MODED.exe but even then the "MODED" game exe will most likely live in the same directory as the original. I think its bad UX to install ReShade DLLs into the custom command directory when only custom command is enabled

I think broadly you might be right, but also, there is a valid use-case for bypassing launchers, which means the custom command would be a game EXE (#968). It can be a bit temperamental (I can't reproduce the bugs in that issue iirc) but it can be done. Usually, the launcher will be in the same folder as a game EXE, but not always - XCOM 2 uses separate folders for its launchers (although whether or not bypassing the 2K launcher for this game actually works, and how many people do this without using the XCOM mod launcher, is debatable).

Basically the exact opposite use-case of this issue is also possible. How often one or the other occurs, I don't know. And this isn't being deliberately awkward I'm only trying to point out there could be other use-cases.

On a technical level, it seems ONLY_CUSTOMCMD impacts this because of the logic in setFullGameExePath. We would probably need to remove ONLY_CUSTOMCMD from this check. This function is only used for ReShade, SpecialK, and something called configureGeoElf. It's probably safe to change.

I feel bad after you put all the work into reshade with custom command (and Im not saying to ditch that we can leave it)

Hey, no need to feel bad :-) We're here trying to figure out the best approach. I'm not trying to shoot you down or anything I'm just putting forward the pros and cons of either solution.

I got a PR up with a tentative fix if you'd like to test it: #1092. I explain the theory in that PR, but basically, in setFullGameExePath (which is what ultimately decides the ReShade installation directory in installReshade) we prioritise the alt EXE path if defined over the custom command EXE path. That means ReShade should use the Alt EXE path instead of the custom command EXE path to install its DLLs to, which should fix the issue here.

This logiic should only apply in the following cases:

When ONLY_CUSTOMCMD is enabled, or
When Custom Commands are enabled + The Custom Command is a valid file + ReShade with Custom Commands is enabled (handles case where Custom Command is launched alongside game and ReShade with Custom Command is enabled)

So I don't think this should cause any regressions, including the 2nd case of launching custom commands with ReShade alongside the game, but let me know if you find any issues. In my head, alt EXE path should be prioritised over the custom command directory if specified, and if a user doesn't want this behaviour, they can disable the alt EXE path.

This definitely needs looked at before UX changes. Putting those changes aside, let's make sure I'm understanding properly: Observed: ReShade DLLs are always put into the folder of the custom command Expected: ReShade DLLs should be put into the folder defined by ALTEXEPATH, which is the behaviour for

Exactly

there is a valid use-case for bypassing launchers

oops didn't think of that

As I said, the expectation should always be that Reshade gets installed wherever their game exe is located.

So really, it just comes down to: should we make the people who are using ONLY_CUSTOMCMD" to launch a custom launcher to load there game enable theALTEXEPATHor the people who are usingONLY_CUSTOMCMD" to bypass the games launcher enable ALTEXEPATH

Either way, someone is going to have the "bad" UX of enabling an extra option. ...So my vote on this then is to leave it alone 😅

In my head, alt EXE path should be prioritised over the custom command directory if specified, and if a user doesn't want this behaviour, they can disable the alt EXE path.

sounds correct to me I'll give it a test now

oh and about specialk yeah I think it may be because of the mod launcher didn't really play to much with it just brought it up as I imagine it should use the same logic as reshade with custom commands?

just brought it up as I imagine it should use the same logic as reshade with custom commands?

It has been a while, but I believe so, they both work on the idea of putting DLLs in the same folder as an EXE so I think a lot of the code is shared and they both use setFullGameExePath to set the directory to put the DLLs (https://github.com/sonic2kk/steamtinkerlaunch/blob/a5984bb1300cf712631c6aa3a4300d4889514d36/steamtinkerlaunch#L10328-L10330).

Either way, someone is going to have the "bad" UX of enabling an extra option. ...So my vote on this then is to leave it alone 😅

That might be for the best, but I will keep this in mind in case it comes up in future. If there are other people voicing similar ideas to yours then we can change it.

should we make the people who are using ONLY_CUSTOMCMD to launch a custom launcher to load there game enable the ALTEXEPATH or the people who are using ONLY_CUSTOMCMD to bypass the games launcher enable ALTEXEPATH

There are arguments for both as we've shown, and while I wouldn't lose any sleep if the default was changed, I might lean towards the existing behaviour. If you were launching a game and wanted this behaviour, you'd have to set ALTEXEPATH, so preserving that behaviour when using a custom command might be very, very slightly more intuitive. Again though, as you've shown the use-cases for custom commands are pretty different, so there could very well be more people expecting ReShade to not be installed for the custom command, and this is an extra step that most users have to jump through, which is the opposite to a game EXE; generally the game EXE is where you want the ReShade DLLs to go by default, which is why it's the default, but this may not be a sensible default for custom commands. If it's not a sensible default for custom commands then that would be the reasoning for changing the UX. It's difficult to know what is the right answer here though, we don't know other than anecdotally what the most common might be.

So I think the idea of leaving it like it is for now is fine, and if others raise issues with this default, or if you have time to think and decide it really doesn't make sense, we can change it. I'm not opposed at all to changing it, and leaving it how it is just means I have less code to write :wink:

sonic2kk / steamtinkerlaunch