Not enough free conventional memory

[usergithub64]

Environment (please complete the following information):

• Windows 10 x64
• DOSBox-X 0.83.1 x64 SDL2

With standard settings, free conventional 640KB memory is not enough.

With the same parameters in the DOSBox SVN-Daum, the size of the free conventional memory looks correct and more believable.

[rderooy]

There are various config options in the [dos] section which effect available memory. If you want more Free conventional memory being reported, you can for instance set

[dos] minimum mcb free=1

And it will report 627K Free.

You don't want the Conventional Free memory to be too high, as otherwise you need to use loadfix for lots of programs to prevent the "Packed file corrupt" and similar errors.

[joncampbell123]

The default minimum mcb free was chosen to reflect what a typical DOS system (6.22 or Windows 95) comes up with when a minimal config is loaded. It was changed to help eliminate the numerous cases where users open issues and report that such and such game crashed or printed "packed file is corrupt" when a simple LOADFIX would fix it.

Be aware that there is lots of DOS software out there that will crash or run erratically if loaded too LOW in memory.

[joncampbell123]

Now another option that should be considered is emulating a configuration where DOS=HIGH,UMB, in which case 630K-634K free would be possible (The DOS is in HMA).

To emulate that would require xms=true, DOS private allocation to allocate from the HMA, and DOS interrupt vectors 20h-21h to enable the A20 gate on every call to emulate actual MS-DOS behavior. Since DOS in HMA requires A20 enabled to execute, HIMEM.SYS on real MS-DOS hooks INT 21h and switches on the A20 gate before passing control to the real INT 21h.

[Dwedit]

Under the default settings, Civilization 1 doesn't have enough memory, and there are no settings in the built-in GUI configuration editor that control the "minimum mcb free" setting.

Personally, I'd put an command on the menus to set the amount of conventional memory available.

[Dwedit]

Update to previous comment:

CIV 1 needs 551K of free conventional memory to run, but won't start on DOSBox-X (triggers LOADFIX) unless 576K or less is available.

Under default settings, DOSBox-X provides 608K of conventional memory. Attempting to run CIV with this much memory triggers running LOADFIX, which allocates 64K of memory. Now it has 544, which is under 551, so there's not enough memory.

Making the change "minimum mcb free = 1" makes Dosbox-X instead provide 627K of conventional memory, and running CIV triggers LOADFIX again, which results in 563K of available conventional memory. That's above 551K, so it runs.

So I think my suggestion would be to change LOADFIX to allocate down to the next 64K multiple instead of always allocating 64K. So 608K Free would allocate until 576K (64K * 9) is available, instead of allocating 64K.

Meanwhile, Dosbox Daum gives no error when running CIV with too much memory, so it doesn't need LOADFIX at all. Dosbox Daum may possibly be emulating a lot less of a DOS system, so that too much memory doesn't trigger an error.

[joncampbell123]

@Dwedit There was a patch submitted a while back where, if you run a program and it prints a common error message given by EXEPACKed executables, the code automatically runs loadfix and runs it again.

I've thought about adding an option to disable that in cases like this, or controlling how much LOADFIX is asked to consume when that triggers.

EXEPACK compressed executables have decompression code with a bug that triggers when the executable image is loaded too low in memory (and the A20 gate is on). The bug causes the decompression code to think the compressed image is corrupted.

[joncampbell123]

Actually... I just thought of something.

@Wengier What if the EXEPACK loadfix hack were modified so that, instead of running LOADFIX and running again, it simply shut the A20 line off temporarily? It could be an option in dosbox.conf: a) do nothing b) run loadfix c) turn off the A20 gate.

The faulty decompression code should work correctly if loaded too low, but the A20 line is off, therefore allowing it to underflow the segment value to FFxx:xxxx but address itself properly (1MB wraparound just like the 8088).

[joncampbell123]

I also happen to know examining actual MS-DOS with HIMEM.SYS loaded that if you loaded DOS into HMA, the INT 21h vector is set to point at a routine that makes sure the A20 gate is on when you call INT 21h and other key interrupts so that DOS doesn't crash when it finally jumps into the HMA where the DOS kernel resides. In DOSBox-X similar emulation will switch on the A20 gate again the instant INT 21h is called.

[Wengier]

@joncampbell123 I think @Dwedit was mainly talking about the behavior of LOADFIX itself, whether or not it is automatically triggered - when LOADFIX is run (either manually or automatically) it always consumes 64KB memory at this time, and as a result there is no enough memory to run the game CIV 1 in the default setting (which I will check soon) which requires at least 551KB free conventional memory as he said.

As for the auto LOADFIX function, there is already a config option to disable this behavior, just by setting autoloadfix=false in dosbox-x.conf. It does not support to change the A20 gate though.

[Dwedit]

Another game to consider is "California Games". Whenever you have more than 615K of free conventional memory, it displays a bogus error message about not having enough memory.

And this game won't even run with more than 576K of conventional memory anyway due to using the buggy Microsoft EXE packer.

Dosbox Daum ignores the bugs in the EXE packer, and can run the game with more conventional memory. If you have more than 615K available, you get the error message: "You are 199K RAM short of having enough memory. You need to buy more memory or remove any memory resident programs, Dude."

So any solution or workaround that bypasses the bugs in the EXE packer (such as the A20 gate thing) and runs the game with more than 576K of conventional memory available should not be a default option, otherwise it would break California Games.

[joncampbell123]

How does Daum "ignore the bugs"? DOSBox in general is an emulator. It emulates the x86 instructions one by one. If the code is faulty in specific conditions then it should fail just like it would on real hardware and real MS-DOS.

The faulty code in the EXE packer fails if the EXE is loaded lower than 64KB from the base of conventional memory, which is why LOADFIX is a remedy for it. My proposal is that the faulty code might happen to work anyway if the A20 gate were temporarily turned off in that case because the 8088-like 1MB wraparound would then work around the bug. In fact it's probably the whole reason IBM added that A20 gate in the first place when first building 286-based PCs.

Apparently it was a big enough problem that many PCs, including clones, and even the NEC PC-98, have a hardware A20 gate and a way to toggle it.

This is what the "minimum mcb" options are for in DOSBox-X. It allows you to control the base MCB segment (base of the MCB chain) and the minimum free MCB block so you can compensate for that.

[joncampbell123]

@Wengier I'll check my theory about the A20 gate and EXE pack. If it checks out, I might submit a pull request that lets the dosbox-x autofix option control whether to do nothing, run LOADFIX, or turn off the A20 gate.

[Wengier]

@joncampbell123 Actually I was looking at this too, and the command A20GATE OFF does allow me to run the "California Game" he mentioned. So with this approach basically the autoloadfix function will turn the A20 gate off, run the program, and after that if will restore A20 gate if necessary. But I wonder if there is any negative effect for running the DOS program when the A20 gate is forced off in this case? I think I will submit a pull request myself then.

[joncampbell123]

@Wengier If DOSBox-X is emulating HIMEM.SYS and DOS=HIGH like it should by default, A20 will be re-enabled on the next INT 21h call. The reason for this is that on real MS-DOS, if HIMEM.SYS is loaded and DOS is loaded into HMA, The INT 21h vector is hooked so that A20 is automatically re-enabled when a program calls into DOS (since, A20 needs to be on for the DOS kernel to run properly).

So the A20 workaround is to call the XMS driver to turn off A20, and then let the next INT 21h call re-enable it again.

[joncampbell123]

If that sounds crazy, note that Windows 3.1 will crash at startup unless INT 21h re-enables A20 at least from my testing.

[joncampbell123]

I'm working on the alternate A20 disable behavior for EXEPACK errors right now.

It would be pretty strange to overload "autoloadfix" to allow other settings like "a20fix". This A20 gate work needs it's own dosbox.conf option.

I'm able to easily test it using the LIB and LINK tools in QuickBasic 4.5 which are apparently packed with EXEPACK.

[joncampbell123]

My change works. Perhaps the option should be called autoa20fix. Then it would have to be clarified that if both autoloadfix and autoa20fix are set, one of them takes precedence.

It instructs DOS to EXEC with the A20 gate disabled (any other INT 21h call reenables A20), which seems to work great.

[Wengier]

@joncampbell123 Great that it works. My original plan is to rename "autoloadfix" to something like "autopackerrorfix", and it will default to "auto", which will auto-decide to use the a20fix approach or the loadfix approach. Alternatively, you could force one of the approaches or disable the fix. I thought that having two config options for the "Packed file is corrupt" error fix might be some kind of duplication.

[joncampbell123]

@Wengier I left it as a pull request and a branch. We could hash out a few details before merging, or we could just merge to master and work it out there.

It's possible the A20 gate fix might work in some cases but the LOADFIX fix might work in other cases.

[joncampbell123]

I'll go ahead and merge to master now, then.

[joncampbell123]

Whoops. There's one more thing to fix:

Forcing A20 off for EXEC is all fine and good for real mode but is a big no-no if the processor is in virtual 8086 mode.

The A20 hack causes Windows 3.1 to crash and the DOS machine to reboot if you run such an executable within a Windows 3.1 DOS VM.

[Wengier]

@joncampbell123 Just a suggestion - perhaps with the "auto" setting it will default the a20fix when in real mode, and will default to the loadfix when in V86 mode?

[joncampbell123]

Even better: I can confirm with DEBUG that Windows 3.1 virtualizes the A20 gate properly, as long as you enable/disable it by calling the XMS driver.

Clearly in DOSBox-X's debugger, the A20 gate is always on, but Windows 3.1's DOS VM knows how to remap the first 64KB past 1MB to make it appear to the DOS VM as if the A20 gate changed.

So the solution is for the a20 fix to run a call to the XMS interface where Windows 3.1 can see it, rather than directly poke at I/O ports.

[Wengier]

@joncampbell123 I see. If a20fix works here too perhaps autoa20fix can always take precedence if both are set.

[joncampbell123]

Hm. I can definitely confirm, even from DEBUG.EXE in a Windows 3.1 VM, that Windows 3.1 does NOT virtualize port 92h, thus the crash if you switch off the A20 gate within a DOS VM.

If it were to virtualize port 92h, switching it off would ask the kernel to remap 1MB+64KB just like XMS would do.

[joncampbell123]

That worked.

Though the INT 21h handler is probably still enabling A20 by writing port 92h, but since it enables A20, it doesn't do anything to crash Windows 3.1. However since it doesn't call XMS, Windows 3.1 doesn't know to virtualize enabling A20 either. Running it again shows no message about reloading.

[Wengier]

@joncampbell123 That is great!:-)

[joncampbell123]

[joncampbell123]

Ha, well, whatever Thexder is doing, it manages to avoid the A20 hack.

It might be something with MAINEG, etc. which are supposedly executable, though encrypted.

This is one case where autoa20fix=false is good to have, so that it falls back to your LOADFIX workaround.

I figured the A20 hack wasn't 100% foolproof. Still, it's a good solution.

[joncampbell123]

The only other test case not to work with the A20 hack is "Space Pigs Megademo". It figures it's corrupt somehow.

Since that's Demoscene stuff I don't expect gamers to find any issue with that.

[Wengier]

@joncampbell123 Yes, I found that too. But running A20GATE OFF before Thexder does work. Not sure why.

[joncampbell123]

The A20 fix only temporarily disables A20. A20 is re-enabled at the next INT 21h call.

The EXEPACK test cases I used don't call INT 21h until after decompressing the executable image, so that works.

A20GATE OFF turns off the A20 gate entirely, which also fixes the issue.

If it turns out the hack needs to avoid other INT 21h calls before re-enabling A20, like calls to resize the main memory block of the executable, then I'll consider how to do that.

[Wengier]

@joncampbell123 For simplicity and consistency I think it can just fall back to the LOADFIX method for now if both are set and the A20OFF method did not work.

[joncampbell123]

Looking at THEXDER.COM INT 21h calls, I can confirm MAINEG and such are encrypted executables. I can see it eventually decrypt the executable in memory.

[joncampbell123]

However MAINEG is executing THEXDER.COM again? That doesn't seem right. Checking now.

[joncampbell123]

Anyway Thexder is happy with the LOADFIX hack.

There was another suggestion that the autoloadfix should not just blindly allocate 64KB, but should allocate however much memory is needed to allocate the lowest 64KB of memory.

I think I'll try to implement that next.

[joncampbell123]

and... it is done.

autoloadfix will allocate only enough memory to ensure the EXE starts at segment 0x1000 (the 64KB mark) or higher. It will not do anything if the memory allocation chooses a memory address at or above 64KB since that would mean the base address wasn't the problem.

[joncampbell123]

One thing I am genuinely interested in as one final autoload hack is to look at whatever patching MS-DOS 5.0 and higher does to EXEPACK executables. As far as I'm aware, there is code in the MS-DOS 5.0 DOS kernel EXEC function to detect the faulty EXEPACK code and patch it somehow.

[Wengier]

@joncampbell123 Well done!

As for MS-DOS 5.0 or higher, I wonder why MS-DOS 5.0+ will still shows the error message "Packed file is corrupt" if it can already automatically detect the faulty EXEPACK code and patch it?

[joncampbell123]

Not sure. Perhaps I'm wrong about the version number. Supposedly MS-DOS 6.2 had it at some point. Maybe it can only detect certain variations of the code.

[Dwedit]

I just tested MS DOS 5.0 on PCEM, and it seems to be able to run packed EXEs with more than 576K of conventional memory free. There is no "Packed File is Corrupt" message.

Again, this is on PCEM.

[Wengier]

@Dwedit Not quite sure, but some packed EXEs may show the "Packed file is corrupt" message only when running in certain environments. For example, you can find another user complaining about getting the "Packed file is corrupt" error after installing MS-DOS 5.0 here:

http://www.verycomputer.com/12_e68dc99b983a6759_1.htm

Also, according to this KB article, the error does seem to apply to MS-DOS 5.0 too:

https://jeffpar.github.io/kbarchive/kb/072/Q72360/

[Dwedit]

Since DOS itself seems to be able to detect some of the buggy packers, I'd suggest detecting the packers in Dosbox as well. Might be a tall order though.

Edit: Is this enough? "01 CD 21 B8 FF 4C CD 21 50 61 63 6B 65 64 20 66 69 6C 65 20 69 73 20 63 6F 72 72 75 70 74" (or in CP437: "☺═!╕ L═!Packed file is corrupt") (edit again: nope, not enough)

For games, it's okay if the program has to run twice, just a bit of annoying red text on the screen while it loads. But this is the kind of thing you wouldn't want on the screen if you were shipping a game on Steam or GOG or something.

But for other programs, or if you are using console redirection, such as pipes or >, you don't want it to run twice.

[joncampbell123]

How about a "autofixwarning" option that controls whether DOSBox-X prints the red text message?

If it's disabled, you can still tell it's happening from the log file.

[joncampbell123]

I also think it might be useful to mod the LOADFIX program to accept a command line option like "-lowmemonly". Instead of allocating 64KB, it allocates just enough to fill the lowest 64KB to avoid the issue (or does nothing if the low 64KB is already allocated).

If you don't want it to run twice, you'd add to your dosbox.conf:

loadfix -lowmemonly game

I could do that right now (in fact I think I will) by copying the code I added to the autoloadfix code in the shell.

[Wengier]

@joncampbell123 Yes, I have already added a "autofixwarning" config option that controls whether DOSBox-X prints the messages when the error occurs in my pull request. Also by setting it to "false" the message Packed file is corrupt will only show up if no fix is applied or the fix(es) failed. You can also use it to individually control the A20 fix and the LOADFIX by setting it to "a20fix" or "loadfix" respectively.

Since you did not appear to have done any modification to the LOADFIX command in the code yet, I decided to go ahead and modify it myself in the pull request, by adding a new command-line option -a (or /a) to LOADFIX command to auto-allocate enough memory to fill the lowest 64KB memory, based on the autoloadfix code in the shell you added earlier. So the command loadfix -a game should work as intended. But if you set the option autofixwarning=false then I don't really see any actual need to do it manually this way though.

[Wengier]

@Dwedit You can just set autofixwarning=false, then you should not notice any thing special on the screen when running such programs.

[usergithub64]

Is there some command that allows a particular game to specify a specific memory size to run? For example, BLOOD game requires 64 MB of memory. And ALADDIN requires a maximum of 32 MB of memory. If there is 33 MB of memory, ALADDIN will not start. Therefore, it would be useful to use a command that, for example, with 64 MB of memory installed in DOSBox-X, would allow the game ALADDIN to run no more than 32 MB of XMS memory.

[Wengier]

@maximus105 You can start DOSBox-X with a command like dosbox-x -set memsize=32 to set the memory size as 32MB. But you cannot dynamically change the memory size when DOSBox-X is already running. Changing memory size certainly requires a restart of DOSBox-X.