search

joncampbell123 / dosbox-x

DOSBox-X fork of the DOSBox project
GNU General Public License v2.0
2.66k stars 378 forks source link

ems=emm386 vs EMM386.exe #2287

Open JayQ2K opened 3 years ago

JayQ2K commented 3 years ago

Getting through some options and trying to get a program to run I found some interesting behavior that I need to check:

  1. When setting ems=true there is no denying EMS gets created when you launch, but VCPI seems to be enabled for some applications and for some not. Now this probably has good reasons and also if ems=true makes ems run in a mode where it's not fully compliant to EMM386.exe this all makes sense.
  2. When setting ems=emm386 (which in my mind should be a full EMM386.exe emulation) you still don't get VCPI the way EMM386.exe gives you. Some applications don't like that they don't get VCPI.

The above to me makes it not really sensible to have ems=true and ems=emm386. Even more having only the ems option available limits you in another way. EMM386 can be run with the NOEMS option. Executing mem will still show Expanded Memory is available, but at this moment it's more XMS that is being represented as Expanded Memory.

My thoughts would therefore be to split some things up: emsmode = normal, emm386, off (normal being ems=true, off being ems=false and emm386 the full EMM386.exe behavior that includes complete VCPI, which is either implied by choosing emsmode=emm386 or requires an added option below) ems = on, off (do you need only XMS or EMS as well; probably default to on) vcpi = on, off (only valid with emsmode=emm386, otherwise discarded) You might include a warning for using emsmode=emm386 + vcpi=on that not all applications have been tested against the VCPI implementation.

joncampbell123 commented 3 years ago

Some programs don't properly check for EMM386.EXE but assume it might be there if they detect the CPU is running in virtual 8086 mode using the SMSW instruction. ems emulation by default does NOT run the CPU in virtual 8086 mode, unlike EMM386.EXE. There is a dosbox.conf option to run the entire DOS machine in virtual 8086 mode with VCPI enabled, but it's not 100% stable and likely to crash the minute any DOS extender tries to use VCPI to switch into 32-bit protected mode.

JayQ2K commented 3 years ago

Figured as much it would be in that direction. Certainly some DOS programs can make interesting assumptions. But looking what you wrote below you would still end up with different behavior than what I figured to happen.

EMM386 doesn't really put the whole "machine" in virtual 8086 mode. The auto switch in EMM386 already puts a stop to that. The config option in dosbox however does, making it effectively impossible to stop running that mode for the current running session if an individual application is better of without virtual 8086.

Also ems=false without EMM386 means really having no expanded memory, while using the noems switch in EMM386 still sort of gives you expanded memory.

JayQ2K commented 3 years ago

@joncampbell123 Just for some clarity (did some additional investigation): xms = true ems = true vcpi = true emm386 startup active = false umb = true

leaves the environment running in "Real-mode" (so not V86-mode) and No VCPI host active, while EMS actually is active including page framing memory (meaning NOEMS cannot be set?)

Is this then some kind of a hybrid situation, where the CPU is switching V86-mode when requested by an application (IRET)? What does that then mean for applications that expect a VCPI host?

"emm386 startup active" can be sort of explained as V86-mode active?

joncampbell123 commented 3 years ago

@JayQ2K Correct. On real DOS systems, loading EMM386.EXE usually brings the whole DOS system into virtual 8086 mode. DOSBox emulates EMM/EMM386 without virtual 8086 mode. DOSBox-X offers "emm386 startup active" to emulate emm386 with virtual 8086 mode, but only if ems=emm386. VCPI emulation is incomplete, so DOS extenders do not quite work with it.

joncampbell123 commented 3 years ago

Believe it or not, one of the reasons "emm386 startup active" and other odd emm-related options were added was the way certain DOS games and demoscene productions detect EMM386. Not everybody had reliable documentation back in the day, so some detect EMM386 by whether the interrupt handler is nonzero. Others detect EMM386 by using the SMSW instruction to detect whether the PE (protected mode enable) bit is set.

Some demoscene stuff is made to refuse to run unless expanded memory is available, but they use the SMSW detect rather than the proper means to detect EMM386, and will therefore fail to run even if ems=true. Those demos need the "emm386 startup active=true" and "ems=emm386" options to run at all. They don't use VCPI or protected mode, they just need expanded memory and the democoder didn't have documentation on how to detect expanded memory availability properly.

Not all demos do the test to require expanded memory. Some demos do their preferred method of EMM386 detection because they cannot run under EMM386 for whatever reason, so the test is to provide a friendly "Remove EMM386 from your startup" message instead of crashing. Usually that reason is related to the "flat real mode" trick to access extended memory from real mode, or because the democoder made their own protected mode environment, neither which is possible from virtual 8086 mode under EMM386.EXE.

joncampbell123 commented 3 years ago

SMSW happens to be a good test for virtual 8086 mode if you're made to run in real mode, because SMSW (80286 store machine status word) is not considered a privileged instruction and therefore can be executed from within virtual 8086 mode and real mode safely without causing a protected mode fault. On the 286, you set bit 0 of the machine status word to enable protected mode. On the 386 and later, the machine status word is mapped to the low 4 bits of CR0 internally for backwards compatibility, therefore protected mode can be controlled from either MOV CR0,... or LMSW, your choice, and can be read using MOV ...,CR0 or SMSW.

JayQ2K commented 3 years ago

@joncampbell123 Interesting reading. Looking a little further the internal XMS misses out on a functionality that might explain a few things on locking up when "emm386 startup active"=true That functionality is 4309h or GET XMS HANDLE TABLE, which is a typical XMS driver 3.1 or newer functionality. Without this available you effectively need to have XMS dynamic memory allocation disabled when activating the EMS driver.

Now dynamic memory allocation is quite significant if you want the EMS driver to be able to address both VCPI and EMM activities simultaneously. EMM386 and JEMM386 figure out automatically if dynamic memory allocation is unavailable and allocate memory at VCPI/EMS initialization accordingly with the EMS in charge of addressing. If the internal EMS always assumes the availability of dynamic memory with "emm386 startup active" the addressing is actually handed over to the XMS. This way you could end up writing to the same memory address multiple times, potentially crashing the system.

JayQ2K commented 3 years ago

Also, with the following setup: [dos] xms = true hma = true ems = true vcpi = true emm386 startup active = false umb = true

and according to comments in the full-reference-config this setup should lead to the activation of VCPI (vcpi: If set and expanded memory is enabled, also emulate VCPI.). So far I can only see VCPI activated when "emm386 startup active" is true. So either the manual should have some updates or something doesn't work according to the manual.

Going through the logic of EMM386; when launching it the VCPI host can only be stopped with NOEMS switched as well. In DOSBox-X it sort of works the other way around where ems is active when switched, but VCPI actually requires more difficult actions.

Wengier commented 3 years ago

@JayQ2K There are many config options in the full config file, and some errors in the comments are certainly possible to occur. If you find mismatches between comments and actual results, please feel free to send pull requests to correct them (either the code or the comments based on actual DOS system behavior). Thanks!

JayQ2K commented 3 years ago

@Wengier: please see below: If the plan was to emulate EMM386 behavior, then the ems and related settings have to be switched almost completely. Under EMM386 you won't get VCPI/EMS unless V86-mode is launched. Without any switches you'll get EMS (with page frame) and VCPI. Use NOEMS to mostly disable EMS. NOVCPI can only be switched in combo with NOEMS. So by then you'll only have V86-mode running without VCPI/EMS. Going over that logic, the manual pages should look something like this:

JayQ2K commented 3 years ago

The above however also means coding changes are required to DOSBox-X memory management. This because emm386 startup active now has become the leading setting in configuring EMS and it is not dependent on whether ems = true.

VCPI only changes in the documentation and is therefore the quickest to do. VCPI could be a machine running option to switch on or off (like ems has) that is greyed-out if emm386 startup active = false.

The ems setting will have a noems setting, in accordance to EMM386 switches. A lot a programs that use a DOS-extender are actually more dependent on VCPI than on EMS, meaning noems actually plays an important role. Besides XMS is faster than EMS (EMS is another layer on top of XMS), meaning programs should respond quicker. With EMS configurable while the machine is running most users should be able to get to their preferred settings without restarting the machine itself.

joncampbell123 commented 3 years ago

Fair warning: VCPI emulation in DOSBox-X is incomplete, which is why the setting is off by default.

JayQ2K commented 3 years ago

Don't think so. Reference config has vcpi = true by default. So the only effective reason why VCPI host is not active is that emm386 startup active = false. But it's not reflected as such in the documentation. VCPI section might need to represent that:

That would also mean that if you want true EMM386 emulation via DOSBox-X including EMS, the VCPI emulation code needs to be verified. In the meantime you will effectively only have default ems.

Wengier commented 3 years ago

@JayQ2K I have checked it myself. With a config file containing only the following settings:

[dos]
xms = true
hma = true
ems = true
vcpi = true
emm386 startup active = false
umb = true

[log]
misc=debug

VCPI does appear to be enabled, which can be confirmed by the line "2437279 DEBUG MISC:Enabling VCPI emulation" in the log console (or log file). Are there other settings that prevent the VCPI from being activated on your system?

JayQ2K commented 3 years ago

With misc debugging enabled I get the Enabling VCPI emulation line too. Thing is that when I run tools that either test VCPI or work with VCPI I get somewhat the following result: "no VCPI host found" (this specific message comes from the VCPI status checking tool in the JEMM package...while JEMM doesn't really emulate EMM386 behavior, the shipped tools are really nice to work with) "No VCPI interface found......"

Changing ems to emm386 and emm386 startup active to true the result is like this (see VCPI.txt attached): VCPI.TXT

Also, when going through the misc debugging log with V86-mode enabled I found this different then without V86-mode: miscdeb.log Deleted everything but the different parts from the logfile. The last line (de08 support) gets swritten when I run VPCI.exe (checked that in the logging console). Found something online for that: https://www.auersoft.eu/soft/specials/emm386/README.txt

The other program I tried in V86-mode crashes the machine with the following: debcrash.txt

So in the end the VCPI host is sort of enabled (the log says so), but unless you launch in V86-mode there is no way for the programs in the machine to know VCPI is actually functioning. So pretty much DOSBox-X is at the moment offering limited VCPI services, but not an actual VCPI host with V86-mode disabled. You are actually required to have V86-mode active if you want to have a full VCPI host available.

Documentation for vcpi switch might now be better like this:

joncampbell123 commented 3 years ago

Right, VCPI emulation is very incomplete at this time. Only slightly more developed than DOSBox SVN as far as I know.

VCPI is tied to EMM386 in DOS.

http://hackipedia.org/browse.cgi/Computer/Operating%20Systems/MS%2dDOS/VCPI%2c%20Virtual%20Control%20Program%20Interface/Virtual%20Control%20Program%20Interface%20%281989%2d06%2d12%29%20v1%2e0%2etxt

JayQ2K commented 3 years ago

Also found this: 2949414 WARN MISC:EMM386 virtual 8086 monitor is not stable! Use with caution! EMS: DOSBox does not support enabling virtual 8086 mode without VCPI.

[dos]
ems                     = emm386
vcpi                    = false
emm386 startup active   = true

Meaning V86-mode in DOSBox-X is dependent on VCPI, while under EMM386 it's the other way around.

joncampbell123 commented 2 years ago

Speaking of the PE bit, there's a question with which processors (Intel or not) allowed SMSW to clear the PE bit.

Intel 286 and 386 programmers references explicitly state that you cannot clear the PE bit with SMSW. The 286 as you know was the one that could enter protected mode but not leave it without a reset.

What makes it murky here is that there doesn't seem to be any reference to the PE bit restriction in Intel references for the 486 and beyond. Did 486/Pentium CPUs at some point allow clearing PE?

There are two reasons to study this related to two DOS programs that seem to expect opposite results.

One is a Demoscene tracker that detects VM86 mode using SMSW. It uses LMSW+SMSW in vm86 in a way that sets the PE bit briefly then expects to clear it soon after. But you're not supposed to clear PE, so the tracker crashes sometime after that point because of the incomplete protected mode state it just turned on. The fix for that tracker was to allow clearing the PE bit. It seems to have been written in the mid 1990s (1996?).

The other has to do with older versions of the DOS4GW.EXE DOS extender, especially that which was "bound" to early 1990s protected mode DOS games like DOOM and HERETIC. Through some interaction with FreeDOS and JEMMEX (the FreeDOS equivalent to EMM386), the flawed DOS4GW code appears to load, mask, and store the machine status word in a way that would clear the PE bit. If clearing the PE bit is allowed, this puts the CPU into an inconsistent sort of real mode that, when the extender later does a FAR call, jumps off into an arbitrary real-mode address and crashes. If not clearing the PE bit is allowed, the DOS extender works and the game runs as expected.

So do Intel processors at some point just give up and allow clearing PE? If so when did that happen? Late 486? Pentium? Pentium II or III? At some point ever? Or maybe Intel never allowed it, but AMD did?

joncampbell123 commented 2 years ago

To explain the old DOS4GW.EXE behavior: For some reason, it stores the MSW into CX, reads 0x0001 from memory into AX, negates AX, ANDs AX with CX, and stores that back into the MSW. The result is 0010h, which would clear the PE bit.

Later DOS4GW doesn't appear to have this weirdness. Perhaps they found out their DOS extended crashed on clone processors or Pentium PCs.

joncampbell123 commented 2 years ago

The weird DOS4GW code and crash in question (CPU log) x.txt

JayQ2K commented 2 years ago

I had to get through this for a bit, but isn't it so that if DOS sets PE active then some applications (Windows can do that as well) can also put it in a kind of stand-by? Otherwise ring 0 is saturated with crashes to follow. From back in the day I seem to remember that older games that required expanded memory never included a way to figure out whether CPU was in protected mode and therefore all those games forced the setting themselves. With the introduction of EMM managers like EMM386 that became a very different thing. In the old days, Windows couldn't start either with PE bit set.

DOS Extenders (at least te later versions) I definitely know to work with the feedback the system gives to figure out PE bit status and then operate accordingly. I don't think btw it was ever possible to really switch out of PE mode, since you would need to completely clear the memory structure and that pretty much requires a reset of some sort.