DFHack console is blank when launched with PyLNP 0.14c on Windows

[lethosor]

I have not reproduced the issue myself because I am on Linux, but several users have reported it at http://www.bay12forums.com/smf/index.php?topic=140808.570 and Discord.

Launching DF manually apparently causes the DFHack console to display normally.

One screenshot from the forums:

[Pidgeot]

There's no special handling of DFHack when launching on Windows, so this is something that's going wrong internally in DFHack, and I can't see any difference that should be affecting them compared with the launch.

I'm happy to tweak the launch process if I'm missing something, but they're going to have to do the investigation, since it's something internal to DFHack that's going wrong.

[lethosor]

I'm reporting this on behalf of the DFHack team. We are pretty sure the issue lies with PyLNP, because we haven't changed the Windows console behavior in a while, and DFHack launches successfully externally from PyLNP. The same DFHack version launches successfully with the "old" 0.14c build, according to @thurin, and other DFHack versions worked with 0.14b and earlier.

[thurin]

I can confirm this behavior. using PE's Starter Packs as the basis. -r09 which used initial 0.14c build works on win10/11. -r10 which uses new 0.14c build shows blank dfhack window.

r09:

r10:

Launching Dwarf Fortress.exe directly and DFHack console displays correctly. It's only when launching from PE's PyLNP that it's blank.

[Pidgeot]

To be clear: I'm not saying it's not happening, I'm saying I can't tell what's going wrong.

core/launch.py itself hasn't changed in 5 years. The only difference between old 0.14c and new 0.14c is that it's built against a different version of Python (some libraries are probably also a different version).

Now, admittedly, since the new one is a 32-bit Python as opposed to a 64-bit Python, this does change a couple of environment variables that get sent in... %CommonProgramFiles% and %ProgramFiles% point to the x86 folders, and %PROCESSOR_ARCHITECTURE% reads x86 instead of AMD64, with an extra environment variable %PROCESSOR_ARCHITEW6432% to provide the "true" value. But, even if I explicitly set these back before launching it, I still see the same issue, even on an old 32-bit DF with DFHack.

So, something inside DFHack is making it think it can't write to the console, but since the console is created and controlled by DFHack in the first place, I have no way of knowing what that difference is, and what I can do to prevent it.

I guess I can try installing a 64-bit version of 3.8 during the weekend and see if that fixes it? But that isn't supposed to make a difference., so I'm kinda grasping at straws here...

EDIT: Forgot to mention that this is specific to the built version, since it works fine when I run with 3.8 from source... so I guess maybe there's something in PyInstaller? I'll see if I can figure something out, but this is just super weird...

[thurin]

Let me know if there's anything I can do to help out, test, etc.

[lethosor]

I'm putting up a test branch at https://github.com/DFHack/dfhack/pull/2359. Right now, it just logs all environment variables to stderr.log on startup (that seemed like the next-best place since the console isn't working). I would appreciate if someone on Windows could run that build under the old + new PyLNP and see if there is any meaningful difference in the logs. Also open to adding anything you think would be useful.

You can download builds on that PR from the "Details" link by the Buildmaster check. The link is going to change for future builds, but is currently https://buildmaster.lubar.me/applications/17/builds/build?buildId=12237

[thurin]

Not much of a diff in the stderr.log.

diff --git "a/.\\PeridexisErrant's Starter Pack 0.47.05-r09\\Dwarf Fortress 0.47.05\\stderr.log" "b/.\\PeridexisErrant's Starter Pack 0.47.05-r10\\Dwarf Fortress 0.47.05\\stderr.log"
index 68e6856..69320e3 100644
--- "a/.\\PeridexisErrant's Starter Pack 0.47.05-r09\\Dwarf Fortress 0.47.05\\stderr.log"
+++ "b/.\\PeridexisErrant's Starter Pack 0.47.05-r10\\Dwarf Fortress 0.47.05\\stderr.log"
@@ -49,8 +49,9 @@ 
 env:VS140COMNTOOLS=C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\
 env:WINDIR=C:\WINDOWS
+env:__COMPAT_LAYER=ElevateCreateProcess
 DFHack build: 0.47.05-r7-62-gff4c081f
-Starting with working directory: F:\Games\PeridexisErrant's Starter Pack 0.47.05-r09\Dwarf Fortress 0.47.05
+Starting with working directory: F:\Games\PeridexisErrant's Starter Pack 0.47.05-r10\Dwarf Fortress 0.47.05
 Identifying DF version.
 Loading hack\symbols.xml ... OK
 v0.47.05 SDL win32 (windows): PE: 0x6011F2EF

Are there any specific ENV that you'd like details on?

[Pidgeot]

It's not related to an environment variable, but I dug a little deeper, and I found out what's causing this.

The input/output handles that get passed along to the process are different: in the version of PyInstaller that I got installed for Python 3.8, PyInstaller hooks the Popen call. Disabling this hook is enough to get it to work.

I also found this has an equivalent bug in PyInstaller already - at least, it sure seems to me like that's the same thing happening: https://github.com/pyinstaller/pyinstaller/issues/7118

I'll see if there's a way I can bypass the hook, but otherwise I'll just have to downgrade PyInstaller.

[Pidgeot]

0.14d has been released, fixing this issue and updating the build documentation to point out this problem.

[lethosor]

Nice find! And since you posted, https://github.com/pyinstaller/pyinstaller/issues/7118#issuecomment-1288215033 seems to me to increase the likelihood it is the same issue. Glad rolling back PyInstaller worked.