Regression with delete _MEI in 6.7.0 (working 5.3)

ManPython commented 6 months ago

Description of the issue

Exist Regression with delete _MEI in 6.7.0 (working 5.3), where _MEI_12345 etc. is multiplied for each app run and note deleted after closing app.

Context information (for bug reports)

Output of pyinstaller --version: 6.7.0
Version of Python: Python 3.11.3 (tags/v3.11.3:f3909b8, Apr 4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)] on win32
Platform: W11x64
How you installed Python: pip
Did you also try this on another platform? Does it work there? - other windows
try the latest development version, using the following command:

pip install https://github.com/pyinstaller/pyinstaller/archive/develop.zip

follow all the instructions in our "If Things Go Wrong" Guide (https://github.com/pyinstaller/pyinstaller/wiki/If-Things-Go-Wrong) and

Make sure everything is packaged correctly

[x] start with clean installation
[x] use the latest development version
[x] Run your frozen program from a command window (shell) — instead of double-clicking on it
[x] Package your program in --onedir mode
[x] Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
[x] Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

A minimal example program which shows the error

pyinstaller --onefile --console script.py
# this same with pyinstaller --onefile --console --runtime-tmpdir _TMPDIR script.py

If _TMPDIR then here is created many _MEI versions..

rokm commented 6 months ago

Does this happen with basic print("Hello world") application? If not, you're going to have to provide minimal example of program that triggers this behavior.

ManPython commented 6 months ago

Not exactly.. right.. simple hello no problem.. Here is "simple code" for this problem.

import code
import sys, os 
import pkg_resources

def Hello():
    print("Hello")

Hello()
code.interact(local=dict(globals(), **locals()))

ManPython commented 6 months ago

Also suggesting to try pack pandas, pyopengl, matplotlip, Pyside6.. just import.. no any call etc..

rokm commented 6 months ago

So how are you exiting the above example? By typing sys.exit() into the interactive prompt or just closing the console window?

Because if the latter, then application has a limited time to clean up after itself before Windows forcefully terminate it.

I cannot reproduce the problem with the above example, because it is probably light-weight enough for temporary directory to be cleaned up in time (even inside my testing VM). But I can imagine that larger applications that contain pandas, matplotlib, PySide6, etc. might be too large to be cleaned up in time.

ManPython commented 6 months ago

Just simple exit() - it's working, and compiled this with nuitka, pyoxy, other tricks as portable. Not seeing reason around how to close, coz exist many handlers for this, I just combined simple basic python app as interactive console, not print as single call.. this representing then spectrum of functions from python, eg. when include pandas.

I also tried build function to clean TEMP and no result.

rokm commented 6 months ago

Do those _MEI directories that are left behind contain all application files, or only some of them?

freddy3333 commented 6 months ago

I hope this isn't the wrong place and I apologize for my naiveite, but I'm an emailproxy user (non-developer) and finding huge numbers of these _MEIXXXXX folders in my Windows 11 user temp folder, which contain huge numbers of what appear to be python-related files/subfolders. I'm also finding that most of them are locked, making it difficult to delete them without the use of an Unlocker utility, which then causes the emailproxy to close.

I don't think I've ever seen a Windows process that creates so much temp data. Is there a simple way to modify the code to either reduce the amount of temp files/folders python (or the emailproxy?) creates, or maybe some type of additional process that could be added either to python or Windows to automatically unlock and delete most of these files/folders?

bwoodsend commented 6 months ago

Sounds like emailproxy is still running (or not shutting itself down properly) if the folders aren't delete-able. PyInstaller itself doesn't do anything to prevent you from deleting its temporary directories but if the process inside them is still using the files then the OS will refuse to let you delete them.

bwoodsend commented 6 months ago

Is this emailproxy thing an indefinitely-running background process? It's probably not a good idea for it to be using onefile mode if it is.

rokm commented 6 months ago

Is this emailproxy thing an indefinitely-running background process? It's probably not a good idea for it to be using onefile mode if it is.

Yeah, it is running in the background (and can be auto-started at login).

It is also using noconsole/windowed mode, so we cannot detect Windows shutdown/reboot/logoff via SetConsoleCtrlHandler, and consequently cannot perform the cleanup. I suppose for windowed bootloader in onefile, we'd need to set up a hidden window and catch WM_QUERYENDSESSION/WM_ENDSESSION event messages. The application itself could also try to handle these events in its code (I see only POSIX-style signal handling there), but I suspect there would still be a chance of race between OS shutting down after child process exists but before the parent process has a chance to finish cleanup.

But really, emailproxy should be using onedir builds instead of onefile ones, since they are distributing as a zip file anyway. Maybe they avoided that due to all the clutter that used to be in the application's directory, but nowadays all that is stored in the _internal sub-directory next to executable, so that shouldn't be a problem. CCing @simonrob

freddy3333 commented 6 months ago

emailproxy has an option to auto-start with Windows, so it's available when Eudora opens. And since most people keep Windows open (I generally keep my PCs on 24/7), emailproxy remains open as well. I've no idea what or when the program generates each _MEI folder, but they remain locked even after Eudora has been closed. So, I'm fairly certain they're being auto-generated by the emailproxy.

I also just noticed that at some point since I shut Eudora down (several hours ago), the emailproxy icon in the Windows task bar vanished. I thought the program might've closed along with Eudora, but there're still two emailproxy.exe processes running via Windows Task Manager.

I just tried to delete one of the -MEI folders, but it's locked. So I manually shut down the two emailproxy.exes via Task Manager and went back to try again to delete the _MEI folder, but it's vanished. So it looks like emailproxy may be deleting its current/last _MEI folder if/when the emailproxy process(s) is manually shut down. And if the program's creating only one of these _MEI folders each time it runs and, then, deletes it when the process(s) is shut down, that wouldn't be a problem.

The thing is that there were 25+ of these HUGE _MEI folders yesterday morning with emailproxy running, but the proxy was NOT actually working—due to a number of configuration issues (solved last night). However, today, it's been running and in use all day....and this may be key....there was only one _MEI folder with today's date. So, just guessing here....perhaps, the 25+ _MEI folders I found this morning were leftovers the program was unable to auto-delete due to the misconfiguration??

I hope this helps and appreciate your input!

rokm commented 6 months ago

In a onefile build, a (different) _MEI folder is created each time the executable is ran (that's why emailproxy should not be using onefile - it just wastes CPU and disk-write cycles on each run). And should be deleted if the program manages to shut down cleanly (i.e., the parent process of onefile build is not forcefully terminated, like it happens during system shutdown/restart).

Those 25+ folders sound like left-overs from previous system restarts/shutdowns. Try manually stopping emailproxy if running, remove all _MEI folders, then restart the system. There should be only one _MEI folder created, for the currently running process. Then restart the system couple of times - you should see that the old _MEI folders keep accumulating, due to reasons outlined in my previous comment.

freddy3333 commented 6 months ago

Bingo! I think that was it.

After verifying there were no _MEI folders in Temp, I restarted emailproxy and ONE folder appeared—with a matching timestamp. Then, I shut emailproxy down via its Quit menu option (which I couldn't do previously, because the emailproxy taskbar icon was gone) and the _MEI folder instantly vanished.

Many thanks rokm!

rokm commented 6 months ago

Hmm, upon some more testing, it also turns out that we indeed have a regression between 5.13.2 and 6.0 - in console-enabled onefile builds, system shutdown/restart events are not properly handled anymore. (note: this has no effect on noconsole-builds of emailproxy, where the issue is what we described above; but the application itself is a nice test case for this, if built locally with console enabled).

That's because #7735 added use of GetWindowThreadProcessId and ShowWindow (which are conditionally called at run-time); these come from user32.dll, and that causes bootloader to be linked against it. This in turn runs afoul of the following caveat for SetConsoleCtrlHandler:

Windows 7, Windows 8, Windows 8.1 and Windows 10:

If a console application loads the gdi32.dll or user32.dll library, the HandlerRoutine function that you specify when you call SetConsoleCtrlHandler does not get called for the CTRL_LOGOFF_EVENT and CTRL_SHUTDOWN_EVENT events.

On way to fix this would be to load user32.dll dynamically and import required functions on demand in the affected codepath.

But I suppose a splash-enabled onefile build will run into the same issue, so I'm going to look into adding that hidden window and handling WM_QUERYENDSESSION / WM_ENDSESSION messages. Which should hopefully take care of both console and noconsole cases...

pyinstaller / pyinstaller