NTVDMx64 incompatible with Windows Subsystem for Linux v2

[kristibektashi]

Pretty sure it's because WSL requires ConhostV2 while NTVDMx64 forces ConhostV1

[leecher1337]

Any idea on how to solve this? It's impossible to get NTVDM work with Conhost > V1, I never worked with Windows Subsystem for Linux (don't know how to even activate it, I guess it's something different than the POSIX subsystem of NT 4). Is it possible to detect whether the launched application is a Linux application, so that the loader could find out and maybe not force ConhostV1 when such an application is launched? Of course this would only work if the Linux application is launched i.e. via explorer, because normal Windows Command Shell still needs to be V1, otherwise it won't be able to launch DOS applications and interact with the parent console. How do you usually launch it?

[kristibektashi]

Here is more information on WSL: https://docs.microsoft.com/en-us/windows/wsl/about

[G-Rumpel]

Hi, I don't have WSL installed but my Conhost is set to V2 by default and I switch it to V1 on the fly only when needed. This is done only for the desired process by starting it through a batch as follows:

...
reg add HKCU\Console /v ForceV2 /t REG_DWORD /d 0 /f
start cmd.exe /c <PATH TO DOS EXECUTABLE>
reg add HKCU\Console /v ForceV2 /t REG_DWORD /d 1 /f
...

It could be the other way round depending on which is more desirable. This is just an idea and as I said I had no WSL installed to test it. Hope this helps!

[leecher1337]

It's hard to find a way on how to solve this kind of issues, because it depends on the usecase of NTVDM.

1) If you are just launching DOS applications by initiating a fresh console, i.e. klicking on a DOS .exe in explorer, the enforcement of Conhost V1 isn't strictly needed in the registry, as Windows is smart enough to allocate a V1 console if a fresh V1 console is needed (on AllocConsole call). In this case, it's enough to restore the original Windows behaviour by:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Console]
"ForceV2"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\ldntvdm]
"ObeyForceV2"=dword:00000001

(The ObeyForceV2 key is needed so that the loader doesn't enforce V1 console, which it normally does).

2) However if you want to open let's say cmd.exe and then start DOS applications from there, you already have a V2 console and an allocated console cannot be converted on the fly, so launchin DOS applications on an existing V2 console will fail (this is the same in Windows 10 32bit, if ForceV2=1, then NTVDM won't work there either). In case I would be able to detect which process is allocating a new console (not sure yet, but I can check), the loader could only force V1 console if - say - cmd.exe is launched and leave V2 console otherwise. But that would possibly limit the use of other applications that also want to launch DOS executables. Or make it the other way round, make a blacklist of processes, i.e. wsl.exe, which do not fall under V1 enforcement, but keep in mind that that would mean that wsl.exe cannot be launched from a cmd.exe console, but only from a fresh console.

3) The most elegant, but hardest solution to it would be to write a ConhostV2 wrapper that tries to hack V1 functions into V2 API. I'm not sure if it's even possible as there also is no source available for ConhostV2 implementation (Microsoft Terminal project on github only contains conhost.exe source, but not the console drivers' source, unfortunately). And I guess the conhost-separation war done intentionally, so ConhostV2 simply isn't compatible with V1, otherwise M$ wouldn't have made different versions for it, but would have just packed it into one backwards-compatible conhost, I guess.

[leecher1337]

It seems that 2. also isn't possible, because of a timing issue: condrv.sys launches conhost.exe (fortunately, it contains code that checks for ntvdm.exe and forces Console V1 there, as previously mentioned). On loading of conhost.exe, AppInit_DLLs, which also load ldntvdm.dll, get loaded after main function that decides which console version to use, already ran, so patches applied by ldntvdm.dll to enforce Console V1 don't have any effect. Therefore we cannot dynamically decide which console version to use. As the condrv.sys is kernel mode, it also cannot be modified for user mode loader.

So I'm a bit out of ideas on how to solve this properly, I guess the workaround suggested by @G-Rumpel seems to be your best bet so far.

[dominicraf]

I just saw this thread. I have always used WSL v1 alongside NTVDMx64 and had no problems with hacking registry keys, so my guess is that the problems only arise with WSL v2. So another workaround might be to use WSL v1 instead. You can google for instructions on how to 'downgrade' WSL from v2 to v1.

[ringtailedfox]

well, i think i screwed up my WSL2 installation by trying to run NTVDMx64... WSL2 just dies whenever i try to run bash (it just says "[process exited with code 1 (0x00000001)]" and powershell (as admin) says it can't run with legacy console enabled.. but it's not enabled... even tried uninstalling NTVDMx64 but that didn't fix it... not sure what else to try aside from re-installing WSL2...

[leecher1337]

Did you manually restore ForeceV2 key under HKEY_CURRENT_USER\Console to 1? Shuold be enough to get back the dreaded V2 console.

[ringtailedfox]

i tried that... i ended up having to uninstall NTVDMx64, rebooted, forced both powershell and cmd to not use legacy console mode, rebooted again, and WSL worked again.

i can try giving it another shot tonight, to see if that works...

also, since i'm on windows 10... what are my options for loaders? will the Win11 loader work?

[leecher1337]

Yes, Win11 loader should also work on Win10. Maybe the recent loader update makes it more useful where NTVDM will at least start with forced V2 console (even though in this case in a seperate window).

[RichieRich711]

Using on Win11 64 and Win10 64 works perfectly!

[kristibektashi]

It's hard to find a way on how to solve this kind of issues, because it depends on the usecase of NTVDM.

1. If you are just launching DOS applications by initiating a fresh console, i.e. klicking on a DOS .exe in explorer, the enforcement of Conhost V1 isn't strictly needed in the registry, as Windows is smart enough to allocate a V1 console if a fresh V1 console is needed (on AllocConsole call).
   In this case, it's enough to restore the original Windows behaviour by:

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Console]
"ForceV2"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\ldntvdm]
"ObeyForceV2"=dword:00000001

(The ObeyForceV2 key is needed so that the loader doesn't enforce V1 console, which it normally does).

2. However if you want to open let's say cmd.exe and then start DOS applications from there, you already have a V2 console and an allocated console cannot be converted on the fly, so launchin DOS applications on an existing V2 console will fail (this is the same in Windows 10 32bit, if ForceV2=1, then NTVDM won't work there either).
   In case I would be able to detect which process is allocating a new console (not sure yet, but I can check), the loader could only force V1 console if - say - cmd.exe is launched and leave V2 console otherwise. But that would possibly limit the use of other applications that also want to launch DOS executables. Or make it the other way round, make a blacklist of processes, i.e. wsl.exe, which do not fall under V1 enforcement, but keep in mind that that would mean that wsl.exe cannot be launched from a cmd.exe console, but only from a fresh console.

3. The most elegant, but hardest solution to it would be to write a ConhostV2 wrapper that tries to hack V1 functions into V2 API. I'm not sure if it's even possible as there also is no source available for ConhostV2 implementation (Microsoft Terminal project on github only contains conhost.exe source, but not the console drivers' source, unfortunately). And I guess the conhost-separation war done intentionally, so ConhostV2 simply isn't compatible with V1, otherwise M$ wouldn't have made different versions for it, but would have just packed it into one backwards-compatible conhost, I guess.

4. ConhostV2 is enforced when you use the Windows Terminal app from the Microsoft store to launch a console app (as opposed to directly launching it) even when the registry key is set to enforce V1 so another solution would be to use the Windows Terminal for things like WSL that require ConhostV2 but have V1 enabled so that you can e.g. directly launch a DOS app from cmd.

[kristibektashi]

Closed because of https://github.com/leecher1337/ntvdmx64/commit/4d67e3f70bf512c0b767652d7f0177bea0c89148