translation of int2f/1687

[stsp]

Hi.

I wrote a new djstub https://github.com/stsp/djstub which happened to depend on that feature. Would be good if it is supported by hdpmi, its very simple to. Test program is attached: comcom32.exe.gz

[Baron-von-Riedesel]

Would be good if it is supported by hdpmi, its very simple to.

If I understood correctly, this API is supposed to switch dynamically from 16-bit to 32-bit ( and vice versa ) while remaining in the very same client? That's something HDPMI isn't prepared to do. Be aware that in hdpmi, there are different binaries for 16- and 32-bit ( hdpmi16/hdpmi32 ).

[stsp]

If I understood correctly, this API is supposed to switch dynamically from 16-bit to 32-bit

Indeed, all translations are implemented for calling from DPMI client that became such from the recompiled real-mode prog. So yes.

( and vice versa )

I don't think the reverse is needed, or even consistently possible.

while remaining in the very same client?

Yes, although it doesn't matter much. Initializing as another client is still fine, except that you will have only one 0x4c termination call at the end. So its simpler to have the same client than to add some termination trampolines.

Be aware that in hdpmi, there are different binaries for 16- and 32-bit ( hdpmi16/hdpmi32 ).

:( Well, it is possible to implement that for separate clients. If they share LDT, the task is simpler. But essentially in the new client you only need to map the memory of an old one to DS and CS, as happens during normal DPMI startup. So it is possible even if they share nothing besides physical memory, but creating the real-mode termination trampoline is hell hackish and difficult (you need to terminate both clients with one 4ch call). I have an example of such an impl, so I know its possible, but at the same time its not an easy task.

[stsp]

Be aware that in hdpmi, there are different binaries for 16- and 32-bit ( hdpmi16/hdpmi32 ).

I reduced that requirement. Attached is the new test-case. Now it is enough to implement 1687 w/o changing the client bitness. It should only reset IDT, create DS/SS aliases with the right bitness and put psp_sel into ES. Would that be possible? comcom32.exe.gz

[Baron-von-Riedesel]

Would that be possible?

I guess yes. But I still have no idea what all this is good for. If the bitness doesn't change, why can't the client "reset" the IDT or LDT on its own - using the standard DPMI functions?

[stsp]

There is no function to reset IDT globally or get the default value for a particular vector. If the program is started under some extender, it just doesn't have an access to such info.

[Baron-von-Riedesel]

Attached is the new test-case.

Well, it's a binary, displaying "DPMI unavailable". This msg is displayed, even if hdpmi16/32 is loaded resident. WTF?

I know that you don't really love writing docs, but don't you think that a few words what the test case expects and is supposed to do would be "helpful"?

[stsp]

Ah, sorry, here's the source: https://github.com/stsp/djstub/blob/main/stub.c#L119 As I said earlier, this prog is compiled from real-mode source, so it has this DPMI switch. It says "DPMI unavailable" exactly because 1687 isn't translated.

[stsp]

One thing to note is to zero out SI register. Otherwise the prog will allocate memory which isn't a DOS memory in that case.

[stsp]

A few more notes.

Even though DPMI usualy inherits IDT from prev client, in this case IDT should be completely wiped, as the prog thinks it starts from real mode.

dosemu2 adds additional flags here:

   /* 32bit DPMI supported (0x1),
    * entering from 16bit-PM supported (0x100),
    * entering from 32bit-PM supported (0x200) */
    _LWORD(ebx) = 1 | 0x300;

You may or may not do the same, no one yet checks those flags. :)

[stsp]

Of course it only adds such flags for a translated 1687, not for a virgin one.

[Baron-von-Riedesel]

Hmmmmmmmm...

If I install a debug version of hdpmi - this version starts to display lots of stuff as soon as a program calls the "initial switch to protected-mode" entry - and then launch your comcom32 ... there appears nothing except that "DPMI unavailable".

I tried to run it with debug, to see what problems it might have with the standard int 2F, ax=1687h, but the program is packed, making things a lot more difficult. I don't think this is a good "test case" for anything...

You know, I'm used to small assembly programs that get instantly to the point :).

[stsp]

The program itself is not packed, but cw32 extender that is run before the program, is packed. :) comcom32.exe.gz But don't worry. Attached is an updated test-case. I added "int3" before calling int2f. Since the program is in C, you'll need to step 20-50 instructions after that int3 to get to the actual int2f call, but it should give you a very good way-mark.

[Baron-von-Riedesel]

Ok, the issue with Causeway has been found: it ignores any installed DPMI host if it detects the cpu in real-mode or finds a VCPI host. To make it actually accept hdpmi I had to boot with Jemm386, install hdpmi and then apply the NOVCPI hack to Jemm386 before running comcom32. comcom32 itself soon crashes with GPF ( writes a file, cw.err ).

[stsp]

The crash happens if idt is not reset.

[stsp]

... or the proper segment bitness in ds/ss is set, or psp not put in es, and so on. You can't expect it to work until the translation is fully implemented. Otherwise I wouldn't be asking for that feature. :)

[Baron-von-Riedesel]

You can't expect it to work until the translation is fully implemented. Otherwise I wouldn't be asking for that feature. :)

Oh... Perhaps I unconsciously expected some error checking ... the int 2Fh, routed to real-mode, shouldn't hurt, since it just returns some values. So perhaps checking the value returned in ES:(E)DI ...

Or do you expect the Int 2Fh actually doing the reset at once? Without calling the proc returned by ES:(E)DI?

[stsp]

No, the call to ES:DI is done. I check AX and CF from 1687 - that should be enough, the spec doesn't tell to check something else.

[stsp]

No, the call to ES:DI is done.

I mean, done after 1687, just as in real mode (that code is supposed to work in real mode unmodified)

[Baron-von-Riedesel]

check AX and CF from 1687 that should be enough,

Well, checking if AX==0 and CF clear, is that reasonable at all? If the call isn't translated, you will get exactly that values, since a DPMI host very likely is installed.

IMO it would be far less hackish if implemented as a vendor API ( int 2Fh, ax=1684h )

[stsp]

1684? http://www.ctyme.com/intr/rb-4534.htm

Well, checking if AX==0 and CF clear, is that reasonable at all? If the call isn't translated, you will get exactly that values, since a DPMI host very likely is installed.

But this is a real-mode prog. The translation service is there to run real-mode progs unmodified, so I don't think enriching the checks is a good idea. I think any missing translation leads to a crash.

[Baron-von-Riedesel]

The translation service is there to run real-mode progs unmodified,

I'm very much afraid that you have to explain your API and the idea behind it in very detail ( not just a link to some abominable GAS-style inline assembly code ), because I obviously have no idea what you're talking about :)).

As you probably know: hdpmi is highly optimised. So any "feature" to be added to hdpmi is only imaginable if it is of "general" interest, that is, a) it cannot be achieved by some ring 3 code, and b) the usefulness is not restricted to just one or two programs.

[stsp]

But what exactly is to explain? The real-mode prog does 1687 and then calls entry point. The translator should allow it doing the same thing, unmodified. Asm code is only calling the entry:

    asm volatile(
        "mov %[mseg], %%es\n"  // put dosmem seg to %es
        "lcall *%[sw]\n"                  // call entry
        "pushf\n"                           // save flags
        "pop %0\n"                       // move flags to a variable
        "mov %%es, %1\n"          // move psp from %es to variable
        "push %%ds\n"                // make %ds and %es equal
        "pop %%es\n"

So I don't really understand what is to explain. There is no "idea", "my API" or alike - just a standard asm sequence to call dpmi entry point.

[stsp]

a) it cannot be achieved by some ring 3 code, and

Real-mode prog doesn't have such means. Unfortunately, in case of ia16-gcc, even the PM-aware prog can't do that because the initial state of interrupt vectors are unknown (they are set by the DOS extender which doesn't save old values).

b) the usefulness is not restricted to just one or two programs.

Take any dpmi-aware real-mode program and run it under the extender. It will do exactly what my example does.

[Baron-von-Riedesel]

The real-mode prog does 1687 and then calls entry point. The translator should allow it doing the same thing, unmodified.

If it's unmodified, why not using the standard DPMI "real-mode" functions for it:

    lea edi, r
    mov r.AX, 1687h
    mov bx, 2Fh
    mov cx, 0
    mov ax, 300h
    int 31h
    mov ax, r.DI
    mov r.IP, ax
    mov ax, r.ES
    mov r.CS, ax
    ; set r.DS/ES/SS:SP
    xor bx, bx
    mov ax,301h
    int 31h

and any special requirements ( like a "fresh" IDT/LDT) are set by calling the vendor API. That's how hdpmi is supposed to work.

[stsp]

If it's unmodified, why not using the standard DPMI "real-mode" functions for it:

I don't understand what do you mean. It is unmodified real-mode code. You instead propose some "modification" to call 1687 via dpmi fn 300, and what does this give, other than the breakage?

and any special requirements ( like a "fresh" IDT/LDT)

There are no "special requirements". When you call the dpmi entry point, it does essentially the same thing! It clears IDT (well, sometimes inherits from prev client, but that's a quirk), it puts psp into %es, and all that! I simply don't understand your point. There is no new API, no special requirements. Its just a casual dpmi entry call. Except that its done from PM, but that's why we have an API translation layer.

[stsp]

If it's unmodified, why not using the standard DPMI "real-mode" functions for it:

Because its unmodified, no? But if we actually do (out of curiosity) the modifications you propose, then the outcome will be:

1. 1687 will return non-zero in SI, which is undesirable.
2. dpmi entry will get invalid mem segment %es because of (1)
3. dpmi entry will create a (new) DPMI client, but we are already in a dpmi client.
4. On client termination, only that new DPMI client will terminate, and the "starter" client will leak.

So its not like things will crash and burn, but there would be a few breakages. Obviously to get things right, you should not do such a modification. You should run the prog unmodified, not break it in a subtle ways.

[stsp]

And there would actually be more problems. Yes, I know that because I did that actually already. :) The main problem is that the new client will get the %cs and %ds regs wrongly: they will map the real-mode trampoline of the dpmi server's 0x300 impl. Instead they should map the prot mode segments or the caller.

[stsp]

And stack will map the dpmi's realmode stack that was used by 0x300.

[stsp]

So when translating eg mouse API, why don't you say, "hey, cheating, there were segments, not selectors, so new API! And the need to set up real-mode callbacks is non-standard! Try 0x300, make sure it breaks, so its definitely a new API!". No, you don't say this all. You just implement a very untrivial translation of mouse API, for a simple reason: it allows the "unaware" real-mode prog to run unmodified under DPMI. So how is that different here? Yes, your proposal to use 0x300 will break, but what does this prove?

[Baron-von-Riedesel]

Yes, your proposal to use 0x300 will break, but what does this prove?

Please don't take my "proposals" too serious, since I'm still trying to find out what you gonna want to achieve ...

Btw, that mouse API translation is something I really hate - and it was implemented for the sole reason to be compatible with Win9X ( which was a strong reason in the 1990s and still is something that matters ).

What I do (supposedly) understand so far is: you want a new(?) program to enter protected-mode ( via int 2F, 1687h ), but the host should not create a new client for this program, but use the "current" one. However, the host should "reset" the IDT/LDT at this occation.

If that's true, I can definitely say that such a feature is way too specific to be of "general interest".

Btw, the DEBUG clone that I maintain does something similiar - it wants to run in the very same client than the debuggee and for this to achieve it hooks int 2Fh and fools the debuggee if it calls this interrupt with ax=1687h.

[stsp]

and it was implemented for the sole reason to be compatible with Win9X

It doesn't matter who else implements what, eg win9x. What matters is that mouse API is used: by all borland tools and by all mouse-enabled progs built with borland. In fact that API may be used by a much more progs, but other extenders would implement these calls on their own, while borland extender passes that to dpmi.

but the host should not create a new client for this program, but use the "current" one. However, the host should "reset" the IDT/LDT at this occation.

You can create the new client if you want. The problem is only that 0x4c will be executed just once. If you can terminate both clients by single 0x4c call - please create a new client then. I had such am impl on github, it was horrible but worked fine. So indeed I do not dictate any rules: this is a standard API so please implement it in any way you want.

However, the host should "reset" the IDT/LDT at this occation.

Just like when the new client is created. If you want - please create a new client.

If that's true, I can definitely say that such a feature is way too specific to be of "general interest".

But any prog that used to manually enter dpmi from real mode, can then work under a DOS extender, entering dpmi as before (except that it is already in dpmi).

Btw, the DEBUG clone that I maintain does something similiar - it wants to run in the very same client than the debuggee and for this to achieve

Its very simple, 0xc00 fn is for that.

it hooks int 2Fh and fools the debuggee if it calls this interrupt with ax=1687h.

What exactly does it do? Maybe it does the right thing? Try to run my example under that debugger, and maybe it will work?

[stsp]

I added the proposal to ia16-gcc to add the needed functionality to avoid the need of using that API from protected mode. If you think such translation is not needed (as it seems you do), then we need to modify both ia16-gcc and, more importantly, the real-mode prog. That way it won't work unmodified. I think its always good to create the translations, allowing more progs to work unmodified, but working in a modified form is better than nothing.