implement HMA-specific int2f 4a03, 4a04

[stsp]

In principle we can load fdpp directly to HMA. This will mean that all freedos-alike relocations should be disabled, and the Dyn heap must be put after INIT_TEXT to not overwrite code. This will mean that INIT_TEXT won't be discarded, but at the same time there will be no need for INIT_TEXT at all. All relocation machinery can then be removed. That would be a huge departure from the freedos architecture, with lots of simplifications and code removals. Whether it worth the trouble, is unclear as of yet.

[stsp]

No, will not run from HMA. I made it so that fdpp is fully contained in UMBs. HMA is no longer needed. So we need to implement this: http://www.ctyme.com/intr/rb-4818.htm and this: http://www.ctyme.com/intr/rb-4817.htm and implement our own mem.exe because freedosish mem can't traverse the HMA mcbs. In fact it doesn't even display HMA stats at all.

@andrewbird is this the structure you asked for in some other thread? HMA MCB layout seems to be properly documented.

[andrewbird]

Is the thread you mentioned https://github.com/dosemu2/dosemu2/issues/1514? If so I think the conclusion I came to was that HMA layout was vendor and even version specific.

[stsp]

Maybe there are some differences, but please see the HMA-specific MCB structure layout here: http://www.ctyme.com/intr/rb-4818.htm

[andrewbird]

That looks like a nice detectable format, better than the DR-DOS one.

[stsp]

So if you add some tests, I can add an impl. Currently things to test are 4a01 and 4a02. 4a0[34] should follow.

[stsp]

Ah I see you mentioned already that this format is different from what DR-DOS has... So you've already seen that apparently.

[stsp]

So yes, looking at your description of DR-DOS format, it seems completely incompatible.

[andrewbird]

okay I'll try to add them later today. I'll probably try them against MS-DOS 7 (aka win95), first.

[stsp]

Thanks! Eventually I am going to take your test code as a base and implement mem from them inside comcom32. Or maybe you can do that too. :)

[andrewbird]

Eventually I am going to take your test code as a base and implement mem from them inside comcom32.

Interesting idea, so what were you expecting the tests to do?: 1/ get the head of the chain and follow it to the end 2/ get free space, allocate new hmcb, see the space decreased, delete hmcb, see that the free space returned.

Firstly, I'd like to extend dosdebug's mcbs command to display hma. I don't think I can call a DOS interrupt (int2f//4a04) from dosdebug to get the head of the chain, or can I? Alternatively is there a known starting address for hma?

[stsp]

Those currently not implemented. We need to test 01 and 02 fns, then implement rest.

[andrewbird]

Not really understanding what's wrong with my first test see t26 branch https://github.com/andrewbird/dosemu2/actions/runs/2906241079 No bootflag set leaving dos, did it work for you?

[stsp]

Thanks! Should now be fixed.

[andrewbird]

Thanks, that looks better.

C:\>c:\hmaspace
INFO: HMA free space == 0xf570
INFO: HMA area at FFFF:0A90
PASS: HMA available

C:\>rem end

Are there no blocks already allocated as I look at FFFF:0A90 in dos debug and see?

dosdebug> d FFFF:0A90

ffff:0a90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0aa0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ab0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ac0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ad0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ae0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0af0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0b00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[andrewbird]

I added the other test, I asked for 35 bytes and I expected the allocation to be rounded up to 48. The response showed 35 bytes allocated.

C:\>testit.bat
testit.bat

C:\>c:\hmaalloc
INFO: Unexpected HMA size allocated(BX == 35)
INFO: Allocated HMA block at FFFF:0A90
FAIL: Test failed

I also looked around the memory address returned and expected to see the HMA MCB, but saw nothing.

dosdebug> d FFFF:0A80

ffff:0a80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0a90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0aa0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ab0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ac0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ad0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0ae0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0af0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[andrewbird]

I also note that the FDPP symbols imported that have HMA in their name aren't in the same segment as the block returned

  fa20:172b     __HMARelocationTableStart
  fa20:1767     __HMARelocationTableEnd
  fb2c:1a80     __HMATextAvailable
  fb2c:1a80     __HMATextStart
  fb2c:2510     __HMATextEnd

Whilst that may be because they are pointers to the real values, it would be nice if there were a symbol to represent the base of HMA so that dosdebug might be able to use it to find the head of the HMA blocks.

[andrewbird]

I ran the same test on MS-DOS 7 (Win95) successfully.

C:\>ver

Windows 95. [Version 4.00.950]

C:\>testit

C:\>c:\hmaalloc
INFO: Allocated HMA block at FFFF:C330
PASS: HMA allocation successful

C:\>rem end

The size requested was rounded up as expected to 48 bytes. Looking at the memory block (-16 bytes) in dosdebug looks as expected.

dosdebug> d  FFFF:C320
ffff:c320 4D 53 82 02 30 00 60 C3 00 00 00 00 00 00 00 00  MS..0.`C........
ffff:c330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:c340 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:c350 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:c360 4D 53 00 00 B0 27 20 EB 00 00 00 00 00 00 00 00  MS..0' k........
ffff:c370 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:c380 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:c390 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[andrewbird]

And if I parse the header structure on Win95

C:\>testit

C:\>c:\hmaalloc
INFO: HMA size allocated 48
INFO: Allocated HMA block at FFFF:C330
INFO: "MS" // signature
INFO: 0282 // segment, our PID = 0282
INFO: 0030 // size(hex bytes)
INFO: C360 // next HMA block offset
PASS: HMA allocation successful

C:\>rem end

If you download the test binaries for Win95 first, you can run it yourself.

(cd test-binaries && wget https://www.spheresystems.co.uk/test-binaries/MS-DOS-7.00.tar)

Then run it as

python test/test_dos.py MSDOS700TestCase.test_memory_hma_alloc

[stsp]

Thanks, should now be fixed.

it would be nice if there were a symbol to represent the base of HMA so that dosdebug might be able to use it to find the head of the HMA blocks.

It will be at ffff:10 so I don't think symbols are needed. Is there no MCB at that address under MS-DOS? Well, if so - there is no such symbol, too. :)

Feel free to push your tests, making some of them MS-specific for now.

[andrewbird]

The start of the mcb structured HMA seems to be slightly different on Win95/MS-DOS 7.xx

win95 FE HMA mcbs seem to start at ffff:0030 win95 SE HMA mcbs seem to start at ffff:0030

I'm thinking for dosdebug's mcbs I could start scanning from ffff:0000 looking for 'MS' at the start of each paragraph. If I find one I could read its .next field and see if that also points to a 'MS'. I think this would be reasonably robust.

Did you structure the HMA mcbs the same as this http://www.ctyme.com/intr/rb-4818.htm for FDPP, only I don't see any 'MS' signatures even though my alloc test was given an HMA block at ffff:0010?

ffff:0000 EA 5B E0 00 F0 30 32 2F 32 35 2F 39 33 F4 FC 00  j[`.p02/25/93t|.
ffff:0010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
ffff:0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................

[stsp]

Only 01 and 02 fns are implemented in fdpp. The rest is TBD. So please proceed with MS-DOS.

[andrewbird]

So should I assume that you'll be using the Win95 format in the future, or should I remove the checks from function 02 that I added to validate the MCB?

[stsp]

Yes, please remove the check. Only 04 is guaranteed to point to an MCB. 02 is not documented that way, even if in MS-DOS it does.

[stsp]

Thanks for tests. When I put DOS=HIGH into userhook.sys, then I see this:

C:\>hmaspace.com
INFO: HMA free space == 0xf570
INFO: HMA area at FFFF:0A90
PASS: HMA available 

Which makes sense. So I suppose HMA is working more or less. We just need to impllement 2 missing funcs and mem.exe.

[stsp]

That means that fdpp can only take advantage of ~2.5Kb in HMA. And that hopefully justifies my decision to abandon HMA completely, getting more pressure on an internal dosemu heap in UMB as a cost. Its now completely filled.

[andrewbird]

I added a further test for int2f/4a03 in my https://github.com/andrewbird/dosemu2/tree/t27 branch. I'm not convinced that DL is used to switch between subfunctions as realloc appears to actually alloc a new block without releasing the old memory. See my output from an enhanced dosdebug mcbs command on windows 95 se.

dosdebug> mcbs

ADDR(LOW) PARAS  OWNER
0215:0000 0x004f [DOS]
  => ADDR      PARAS TYPE USAGE
     0216:0000 0x000a [D] Driver (EMUFS   )
     0221:0000 0x0022 [I] Installable Filesystem (UMB     )
     0244:0000 0x0020 [B] Buffers
0265:0000 ------ [LINK]
026a:0000 0x0004 [FREE]
026f:0000 0x0041 [FREE]
02b1:0000 0x9d4d [COMMAND - Data] (END)

ADDR(UMA) PARAS  OWNER
9fff:0000 ------ [LINK]
c300:0000 0x019a [DOS]
  => ADDR      PARAS TYPE USAGE
     c301:0000 0x0009 [D] Driver (EMS     )
     c30b:0000 0x0031 [D] Driver (CDROM   )
     c33d:0000 0x0039 [I] Installable Filesystem (driver i)
     c377:0000 0x0082 [F] Files
     c3fa:0000 0x0010 [X] FCBs Extension
     c40b:0000 0x008f [L] CDS Array
c49b:0000 0x1b63 [FREE]
dfff:0000 ------ [LINK]
f000:0000 0x0165 [COMMAND]
f166:0000 0x0081 [COMMAND - Environment]
f1e8:0000 0x0817 [FREE] (END)

ADDR(HMA) PARAS  OWNER
ffff:0030 0x0321 [MSDOS.SYS]
ffff:3250 0x09f6 [IO.SYS]
ffff:d1c0 0x0025 [fffb]
ffff:d420 0x00b8 [026e]
ffff:dfb0 0x0003 [028d]
ffff:dff0 0x00ad [FREE]
ffff:ea70 0x0005 [028d]
ffff:ead0 0x0002 [028d]
ffff:eb00 (END)

[stsp]

Where have you found about realloc? I think 0 means alloc and some other value (1?) means free?

[andrewbird]

It's on the RBIL page http://www.ctyme.com/intr/rb-4817.htm, but very poorly formatted. It seems to describe alloc 0x0, resize(which I called realloc) 0x1, and asks a question about dealloc 0x2. To be honest the RBIL description page looks unfinished. However for me both DL=0 and DL=1 seem to alloc, and DL=2 does nothing (but that may be because I didn't supply all the other parameters correctly for it to alloc a block). I think I'll have to try tracing into the interrupt with dosdebug to see if I can find what it is looking for, unless you have any better ideas?

[andrewbird]

Okay so a little update: 1/ Interrupt code does switch subfunction on DL as documented. 2/ Dealloc(free) doesn't return any success/fail value, but if you check the id segment in the block you tried to free you can see if the call was successful. 3/ Dealloc(free) does work, but only if the block has not been resized between alloc/free.

So further work is required to see why resize is creating additional blocks, and preventing final free.

[stsp]

Resize usually creates the extra block when shrinking. Shouldn't when growing.

[stsp]

https://github.com/dosemu2/dosemu2/runs/8061348149?check_suite_focus=true Tests failed w/o any changes from me. What can that be?

[andrewbird]

It's the scheduled full test which includes FreeDOS 1.2. Currently that doesn't round up the allocation to paragraph size, so will need to be excluded from the test run.

======================================================================
FAIL: Test FR-DOS-1.20 Memory HMA allocation                                          
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_dos.py", line 3296, in test_memory_hma_alloc
    memory_hma_alloc(self)
  File "/home/runner/work/dosemu2/dosemu2/test/func_memory_hma.py", line 111, in memory_hma_alloc
    self.assertIn("PASS:", results)
AssertionError: 'PASS:' not found in 'testit.bat\r\n
C:\\>c:\\hmaalloc\r\n
INFO: HMA size returned 35\r\n
INFO: HMA block allocated at FFFF:B6F8\r\n
WARN: HMA returned size unexpected\r\n
FAIL: Test failed\r\n
C:\\>'

[andrewbird]

I already have a patch for that test in my t27 branch to avoid this problem https://github.com/andrewbird/dosemu2/commit/9279bbc43a4f9726f569b796a4bb334332b81340

[stsp]

Cool, please make a PR? :) Btw, I think freedos probably actually does the roundup. But it returns the amount you requested, rather than the one with the round-up. At least that way I recall the code that I recently removed from fdpp.

[andrewbird]

Done.

[stsp]

All functionality is added. But I don't suppose anything works. I simply adapted the memmgr.c code for HMA, but it required lots of small and nasty changes everywhere. So that code is broken by default, and we need to add tests and fixes. Andrew, would you like to do that? :)