v6.10.0 `_MEIPASS2` in onedir application

[arossert]

I have an issue with the latest release which might not be a bug but the behavior was changed, I have a weird use case (hard to explain) when I use onedir application that in some cases needs to be executed from a different location, what I'm doing now (until v6.10.0) it copying only the .exe file to the new location, set the _MEIPASS2 env var to point to the original application directory and execute the new file from the new location.

Up until the latest version (v6.10.0) this worked without any issues, not with the changes in #8634 this stopped working, I tried to set the _PYI_APPLICATION_HOME_DIR env instead of the _MEIPASS2 one, set PYINSTALLER_RESET_ENVIRONMENT but nothing worked.

The error is that it tries to load the python38.dll file from the current directory instead of the original one, if I change the CWD of the new process it works but in some cases I need it to be the new directory.

As I mentioned, I know that this is a weird use-case but you can retain this flow with the new mechanism it will be awsome

• Version of Python: 3.8
• Platform: Windows

[rokm]

_PYI_APPLICATION_HOME_DIR (former _MEIPASS2) is now used only by onefile builds to communicate the location of (temporary) top-level application directory, while onedir builds now always determine top-level application directory location based on the executable location.

Even if _PYI_APPLICATION_HOME_DIR was honored in onedir builds, you would also need to fake _PYI_ARCHIVE_FILE (otherwise we would detect that different executable/archive file is used, and reset environment variables - same as if you set PYINSTALLER_RESET_ENVIRONMENT). And doing so would also lead to incorrect process type classification, but that might not be a problem in Windows and in your use case.

[arossert]

I also tested it with v6.9.0 and v6.8.0 and it seems to have the same issue, it does work on v6.7.0 so it is not a new change (I upgraded from 6.7.0 to 6.10.0), I'm not sure but it may be this change #8569

Can you consider enabling this capability back in a new version?

[bwoodsend]

Can you consider enabling this capability back in a new version?

You'd have to give an insanely good reason why moving the executable is a good idea. It was never a feature – just a hack that somehow used to work. I'm amazed that you got away with it this long.

[arossert]

I know that this is a hack a weird use-case as I mentioned, we need it to run from a different location and we don't want to copy the entire directory, this will help us a lot if this will be available even as an "internal hack" as it was before 😃

[rokm]

I also tested it with v6.9.0 and v6.8.0 and it seems to have the same issue, it does work on v6.7.0 so it is not a new change (I upgraded from 6.7.0 to 6.10.0),

True; onedir applications becoming invariant to _MEIPASS2 was probably already brought about during bootloader refactor in #8557.

I suppose we could add a "public" PYINSTALLER_OVERRIDE_APPLICATION_HOME_DIR environment variable that lets you override/specify onedir's application's top level directory (i.e., a branch here). It would be a lot cleaner than trying to resurrect the old behavior via internal variables, as it would (should?) not interfere with other mechanisms that we now have in place.

[arossert]

@rokm If this is not too hard and it can be implemented it will solve my issue 🙏

[bwoodsend]

I'd still like to hear why moving the executable is necessary and to see if there's a better way to solve the problem that it's solving before we lock ourselves into maintaining this esoteric runtime mode forever.

[arossert]

@bwoodsend This is a very specific use case for our product, our application is designed as a 2-process application, the main application spawns child processes runs some logic, and closes it, in some cases the child process needs to be executed in the user context directory (usually APPDATA), in this case, we dynamically copy only the executable file, execute it, and delete it at the end.

The child application is bundled as a onedir application since this happens a lot and we did not want to extract the files each time.

Internally we knew that this day may come as we used an internal property (_MEIPASS2) in order to work around this issue 😃

If this can be supported officially it will help us a lot, I'm willing to contribute a PR if needed and help to maintain this flow as much as I can.

[rokm]

@bwoodsend This is a very specific use case for our product, our application is designed as a 2-process application, the main application spawns child processes runs some logic, and closes it, in some cases the child process needs to be executed in the user context directory (usually APPDATA), in this case, we dynamically copy only the executable file, execute it, and delete it at the end.

But why does it matter where the executable is launched from? Why do you need a copy of it in the user context directory in the first place?

[bersbersbers]

But why does it matter where the executable is launched from?

The only reason I can think of is "working directory", ensuring that a user double-clicking the executable ends up in the right folder when opening a file from within the app or referring to relative paths. But that's nothing a shortcut couldn't take care of, too. And if it's not for GUI users, setting the working directory to %APPDATA% (instead of copying the file) should work just bad well.

[arossert]

@bersbersbers @rokm I'm working for a cyber security company so it's hard to explain the use case without disclosure too much.

All I can say is the user context is very important for the product and this is why we chose this path, we want to avoid copying the entire directory as our application is very large with a lot of dependencies, this is why the _MEIPASS2 usage was perfect for our usecase.

And I forgot to mention that this is a console application, no UI is needed in this flow

[bwoodsend]

Without being able to verify that this isn't a misguided workaround to a problem that could be solved in a more normal way then I'm pretty set towards a no. Even if it's easy to do, it'll still need tests, another minute or so of CI/CD time per platform and will get in the way of future changes in that area. We'd also be unable to check whether your company is still using it or we're carrying this code around for no reason.

[arossert]

@bwoodsend I get your concerns, exposing the ability to place the executable in one directory and the actual libs in another give a lot of flexibility for the end user to change the application directory structure when packing the application even in onedir mode the same as you added for onefile.

The reason we need it related to simulate malicious activity on the host (this is why the location of the executable is important)

For now we are forced to fix PyInstaller version to 6.7.0 until we can find a workaround or we can work with you to make it officially supported, even if it will be implemented as an internal feature as before and will not be covered in any test or CIs (we have internal testing and this is how we found out it is not supported anymore).

I hope we can work together to find a solution, the worst case for us is to maintain a fork with this capability enable which I prefer not to do as it takes a lot of resources form the team.

[arossert]

@bwoodsend I would like to re-open this discussion as we tried several workarounds and nothing worked, for now, we pinned the PyInstaller version to 6.7.0 but we want to keep getting updates and not stay behind with the code.

Is there anything you think you can do to make it also work for onedir and not only for onefile? 🙏

[rokm]

Is having a private fork with the modification described in https://github.com/pyinstaller/pyinstaller/issues/8734#issuecomment-2286528679 not an option? After all, you are after a special behavior that makes little sense in the general context of onedir builds...

It is not a horrible hurdle from technical point of view (at least at the moment, with the way the affected part of codebase is organized), but I do not find the concept of "detachable" onedir applications particularly useful for PyInstaller in general; it requires an environment variable, which cannot really be set "globally" (because there might be multiple different PyInstaller onedir applications running on the system), so it would likely require an additional launcher script (which kind of voids the idea of having executable in one place and contents directory in another, as you could just place the launcher script in the first location and have the complete application bundle in the second one).

Is there anything you think you can do to make it also work for onedir and not only for onefile? 🙏

Just to make this crystal clear, if it "works for onefile", it does so because you are messing with what are now documented internal/private environment variables, which is warranty-voiding (well, except there is not warranty in the first place).

[bwoodsend]

I don't think it's fair for us to have to carry such an unorthodox feature around forever just for one user.

If you feel that your use case is so important that it justifies pushing this unwanted feature on us it shouldn't be much of a stretch to justify maintaining your own private fork.