help wanted: Windows 3.1 Win386 vxdcall header development

[joncampbell123]

There are a lot of calls in the Win386 VxD world in Windows 3.1 (and many more in Windows 95, 98, ME).

I have simplified entering calls by making a *.vxddef list that you compile into a header. It makes it easier, but there is still a lot to be done.

All you need to help with this is a DOSBox-X setup to run Windows 3.1, and a copy of the Windows 3.1 DDK.

Mount the ISO image within the DOSBox-X virtual machine (or unpack it into a directory, whichever is appropriate for your setup).

Bring up Windows 3.1, then use the Run command to run WINHELP.EXE. Go to File, Open, and then navigate to the drive letter or folder the Windows 3.1 DDK exists. Go into the DOCUMENT directory, and then open VDAG31WH.HLP. You'll know you have the right HLP file if the title is the "Windows 3.1 Virtual Device Adaptation Guide".

Going down the list in windows/w9xvmm/dev_vxd_dev_vmm.vxdcalls, look up each call by name using the search function, and then use the documentation provided to enter the input and output parameters of the call.

There are two major types of calls: ASM-like calls and C-like calls.

ASM-like calls are defined in terms of filling in registers before the call, and pulling values out of registers after the call. C-like calls are defined as one where you push parameters onto the stack and then get a return value in EAX (or sometimes EDX and EAX).

[joncampbell123]

The goal of these headers is to make it possible to write Windows 3.0/3.1/95/98/ME 386/VxD device drivers using open source tools.

The tools of choice are:

Open Watcom 2.0 linker (wlink) GCC 6.1 or higher (The C compiler of the GNU Compiler Collection)

[watlersworld]

Thanks

[joncampbell123]

No problem!

The open source world has made cross-compiling to win32 and win16 possible, but so far has not done anything to make .386/.vxd development possible.

Let me know if you have any contribution I should merge into this project!

[watlersworld]

Can Open Watcom C/C++ produce operational TSR drivers that do not crash win16?

[joncampbell123]

What exactly do you mean TSR?

Do you mean a DOS program that terminates, but stays resident, and runs underneath Win16? Or do you mean a Win16 task that runs in the background?

[joncampbell123]

My experience with Open Watcom C/C++ is that while they provide a C function to terminate and stay resident, they don't explain how to pass into the function the number of pages (16-byte blocks) to keep resident. Any crashiness you have had in the past may have been specifying too few pages to keep resident. I get the impression they only intend for .COM executables to use it.

If you want an example of how to compute the correct number of bytes, look in DOSLIB under hw/vga/vga240.c (a TSR that forces all standard VGA modes to 480-line resolutions to improve capture with scan converters, capture cards, or even YouTubers like danooct1 who point a videocamera at the monitor). I do some hackery there to terminate correctly as an EXE.

[watlersworld]

After reading about *.VXD files I am more uncertain than ever. In order to operate with Standard mode windows a TSR must provide win16 driver support. But Microsoft says that the VXD is far better.

I see that each VXD consumes resources and slows down older computers. Also,when VXD activity is too high windows crashes (on real computers). :(

Under Enhanced mode win16 programs can already load 32-bit code. What are *.VXD files best used for?

[joncampbell123]

User-space Windows 3.0 and 3.1 is entirely 16-bit (inherited from earlier Windows that literally ran atop MS-DOS). There are a few exceptions for Windows 3.0 and later, 386 enhanced mode, where code can create 32-bit code and data, and Windows 3.1 Win32s has a driver to create 32-bit code and data Win32s segments when you run a Win32 application.

In Windows 3.0 and Windows 3.0/3.1 286 standard mode, that 16-bit userland is combined with a 16-bit DPMI server. All driver code runs from userspace, and interacts with interrupts and resources (just like you do if you were an MS-DOS application) except usually from a DLL, with consideration for segments that can be discarded or kept resident.

In Windows 3.0 real mode, you're just running atop MS-DOS, and Microsoft's userland API.

In 386 enhanced mode, userspace is still 16-bit, but the kernel in ring 0 runs entirely in a flat 32-bit memory model. Device drivers written for this mode are 32-bit code and are compiled into a variant of the Linear Executable format set up by Microsoft to become the .386 and .VXD files you see all over Windows 3.x/9x/ME. Each file is a "virtual device" that Windows loads into memory. Virtual devices then call one another (and the core kernel) using "services" or a special encoding of INT 20h that carries a virtual device and service number. These services are the API through which a VxD interacts with the system, and the userland underneath.

The reason they are "virtual devices" is that the majority of code (Microsoft and otherwise) assumed the MS-DOS style of development where you either call a library or talk to hardware directly. In fact, if you write a Win16 application, you can code that application to poke at hardware just like you do from MS-DOS and it will work the same (PC speaker, DOS/BIOS calls, anything) so long as you consider you're running in an environment where other Win16 code might be doing the same thing. I/O ports are left open, rather than closed up as in Windows NT.

Some good examples of this, from software I've tested in DOSBox-X so far:

• WinNuke (PC speaker "music" is done by talking directly to the PC speaker from Win16)
• An old copy of Cakewalk for Windows 3.1 (the "metronome" is, again, done by talking directly to the PC speaker since nothing in the documented Win16 API provides anything beyond making a simple "beep")
• The old "PC speaker" sound driver that used to be popular (Win16 code, written as a sound card driver, driving the PC speaker to produce something resembling audio).
• Norton Backup (talks DIRECTLY to the floppy controller to drive a QIC tape drive, because that's where you connect the tape drive when physically installing it).
• Video for Windows AVI playback (when you play an AVI fullscreen, it uses DISPDIB.DLL to kick aside the Windows GDI and calls down through the DPMI server to setup INT 10h mode 13h 320x200x256-color VGA mode to act on the media player's behalf. It also contains code to further reprogram the VGA card for 320x240x256-color using very common "Mode X" programming techniques if the AVI file is 320x240 rather than 320x200).
• Speaking of DISPDIB.DLL, there is proof of concept code in DOSLIB that abuses DISPDIB.DLL to then go on to use other BIOS INT 10h video modes, or reprogram the VGA hardware further, even from a 32-bit application in Windows 95 through ME. :)
• Windows 3.1 multimedia functions (some functions like timeGetTime() in Windows 3.1 use the system "timer", which provides only 55ms resolution, and then talks directly to the timer chip on I/O ports 40-43h to refine the current time in order to provide you with millisecond precision).
• MOUSE.DRV, the mouse driver for Windows 1.x through Windows 3.1, is a user-space DLL that contains code to talk directly to the serial port, or to a bus mouse, or call directly into the BIOS (INT 15h) to initialize and setup a PS/2 interrupt callback into the DLL. In 386 enhanced mode, there is a VXD to virtualize the keyboard and PS/2 mouse only as far as it needs to make this work while isolating from DOS box VMs in the system. This is according to the source code for that driver on the Windows 3.1 DDK CD-ROM.

To put it another way, so much of early Windows 2.x-3.x was evidently written around the MS-DOS model of talking directly to hardware that Microsoft decided it was easier to model the 386 enhanced mode kernel around "virtualizing" hardware rather than provide solid APIs to control the hardware. Thus, VXDs are "virtual devices" that control the hardware, and then use I/O trapping on the 386 to virtualize the hardware so MS-DOS programs and Windows itself can continue poking directly at hardware without conflicting with each other. A proper API for talking to system hardware didn't appear in the kernel until Windows 95.

Keeping userland 16-bit allowed Microsoft (at least for awhile) to make Windows 3.1 32-bit without having to rewrite much of their code at the time, and allowed the desktop to run on both 286 and 386 processors, until they eventually dumped 286 supports and went on to make Windows 95.

[joncampbell123]

If it helps, think of Windows 3.0 and 3.1 as Microsoft's userland and kernel sitting atop a DOS extender, and then using the DPMI functions to call down into DOS and BIOS interrupts where needed while also taking on hardware level programming atop it. Much like a DOS game does with a DPMI server, except with code to separate userspace and kernelspace. If you dig down into Windows 3.1, it's basically both VxD drivers and userspace running in an environment where they are still heavily dependent on the DOS environment below.

[joncampbell123]

To clarify about MOUSE.DRV in Windows 3.1, the driver literally talks to the serial port and bus mouse directly, or for PS/2 mice, is 100% dependent on BIOS INT 15h functions to hook the PS/2 interrupt and receive mouse input via callback. Just like you would an MS-DOS mouse driver, but written to interface with Windows.

[watlersworld]

I see that the Real inheritance stems from ancestors and that display,mouse,keyboard,sound,serial and all Win16 programs have hardware access. What hardware is this 32-bit VXD hokey-pokey best for?

[joncampbell123]

In 386 enhanced mode in Windows 3.x, it's to virtualize the hardware for cooperative access across VMs (DOS boxes as well as Windows itself). In Windows 95 and later, writing a VxD is a necessity for device drivers because most of the hardware access and low level APIs have moved into the kernel level.

VxD development is not specific to any hardware, but a necessity to work with the Windows kernel to get done what needs to be accomplished.

You'll notice the 286 directory has user-space code to poke at hardware, then the 386 directory has virtual drivers to virtualize the hardware.

For example, VGA support has a user-mode component in 286/display/4plane, and a 386 enhanced virtual driver in 386/vddvga to virtualize it (including anything related to VGA such as trapping INT 10h calls). Notice the 386 driver does only virtualization, and very minimal VGA reprogramming, so the 286 VGA driver can do all the work from userspace and share the VGA hardware with DOS VMs.

VxD development for hardware in Windows 3.x is basically virtualization of the hardware to allow user-space to "get along" with other virtual machines, rather than putting all the device driver functions in kernel-space. How you do it varies from driver to driver. Putting actual driver functions in the kernel didn't happen until Windows 95, but since those are also VxDs this development project would be useful there as well.

[joncampbell123]

Oh cool!

I just found an even bigger example of the minimal separation in Windows 3.1 between user and kernel space.

Check out the 8514 VGA driver, 286/display/8plane/8514/source/data.asm:

Get386Procs proc    near
;--------------------------------------------------------------------
;Call into the VDD and get the Ring3<-->Ring0 entry points.
;If the entry point does not exist, then fail (with carry) and
;old BitBlt code (w/o ring transition code) will be loaded instead.
;
;Entry:
; si: 200 or 220
;Exit:
; If (no carry)      //success 
;   si = 202 or 222
; else               //fail
;   si = 200 or 220
;--------------------------------------------------------------------
    push    es
    add si,5            ;205 or 225
    xor di,di
    mov es,di
    mov ax,GET386API
    mov bx,VDD
    int MULTIPLEX
    mov ax,es
    or  ax,ax
    jz  short G3P_NoVDD     ;will be zero if no VDD is present.

    mov word ptr lpfnVDD,di
    mov word ptr lpfnVDD+2,es

That's right: the VDD (virtual display driver) provides a user-mode API entry point to call user-mode subroutines (from ring 3) in ring 0 and vice-versa.

I'm gonna have a lot of fun with this one for future DOSLIB development, especially if this secret API is carried forward into Windows 95 and later.

Still though, development would be cleaner by writing device drivers directly instead of hacking into kernel mode and hardware from user-space.

[watlersworld]

Then virtual machine services are for legacy applications and would mostly be for multitasking of DOS applications. When I consider running more than just one active DOS VM , I know that legacy hardware is out. The VXD must be for high speed interrupts and bus mastering.

Is ring0 just for VM management also?

[joncampbell123]

No. The Windows 3.1 desktop itself is a VM. It's the "system VM", but it's a VM nonetheless. The virtualization that applies to DOS applications, also applies to the Windows 3.1 desktop as well.

Since Windows 3.1 calls down to DOS anyway, it makes sense to mostly treat the Windows 3.1 desktop as just another DOS VM that's fully aware it's running in Windows.

At the time Microsoft made this up, there was only the ISA bus and DMA. PCI didn't exist yet. In the VxD world, there is an API for VxDs to control DMA. For user-space, the DMA controller is virtualized by a VxD that controls the DMA controller. VxDs are ideal for interrupt handling because ring-0 handles it directly. If a VxD does not hook a hardware interrupt, then the Windows kernel by default reflects it to the currently active VM (so, some overhead related to 386 protected mode and user-space). There are also VxD service calls that can be made where a VxD can trigger virtual hardware interrupts in a VM (even if the actual hardware didn't fire anything) as part of virtualization.

[watlersworld]

Humm... a lot to think about there... Is 32-bit code loaded by a 16-bit application run in the same VM?

While win/s Windows covers a theoretical 512mb of RAM , under Enhanced Mode this was reduced to 256mb. Is the lost memory still managed for the Virtual machines?

Do you think deep slot ISA or MCA might have supported bus mastering?

[joncampbell123]

As far as I know, the win386 kernel provides and uses DPMI, which means anything running under it is capable of creating 16- and 32-bit segments (though not ring-0 of course). MS-DOS applications in v86 mode are permitted to switch into protected mode.

Most commonly, MS-DOS applications through a DOS extender will use DPMI to run 32-bit code.

There is ONE example I know of where 16-bit Windows code creates and executes 32-bit code and data, and that is Watcom's win386 extender, which uses DPMI to create a small 32-bit execution environment (with "thunks" to call down to and accept callbacks from Win16). Later on, Microsoft created the Win32s subsystem, which uses a VXD driver and 32-bit DLLs to create a 32-bit Windows application environment atop Win16.

MCA and EISA (deep slot ISA) do support bus mastering. I have no idea whether Windows 3.1 even supported those busses. If it did, it was probably through third party drivers included with the device driver to talk through it.

I'm not sure what limited Windows to 512MB of RAM, but it's probably not the architecture so much as something within the memory management code. I don't believe 386 enhanced mode is any further limited, since I've successfully run Windows 98SE with 512MB of RAM.

[watlersworld]

Thought I saw a Win16 ring0 example once. http://www.kay-bruns.de/freeware/index.html

I believe the WING kit has a win16 example of loading 32-bit code.

Windows 3.1 seems work OK with 64-bit PCIE bus mastering hardware. Video: https://github.com/watlersworld/DIBDRV Audio: http://turkeys4me.byethost4.com/files/Hdadrv5.zip

[watlersworld]

Virtual devices are great as long as they do virtually nothing and pass on the hard work to hardware.

Virtualization made Enhanced mode powerfull- y slow on 386/486 hardware. Do all VXD files require Pagefile Memory?

I know that Far pointers and functions are always slower than Near ones within a single VM. Do VM to VM calls increase call time once again?

A few win16 CODECs have their own *.VXD files. Do you think that a good practice?

[joncampbell123]

Well, VxDs typically virtualize things, but in general, they're drivers that interact with the Windows kernel, and that's what's important. VxD development becomes a lot more important in Windows 95 because what was once done in user-space gets moved into the kernel.

VxDs do not require paging, though there is support in the LE structure to mark "segments" as fixed or pageable, so portions of the VxD that don't involve interrupts can be paged to disk as needed.

VM to VM calls require a task switch. If you're talking about VxD to VxD calls (INT 20h with service+device WORD pair), those are slow to execute the first time because Windows has to search the VxD list, but, to speed things up, Windows will then patch over the INT 20h instruction with a CALL FAR instruction that calls directly to the VxD service call. If execution passes that call again, it's as fast as a direct CALL to the other VxD.

FAR pointers are only a concern in the user-space part of Windows 3.1, since that is normally 16-bit. In the VxD environment at ring-0, all pointers are flat 32-bit except where the VxD has to concern itself with the segmented 16/32-bit world below.

Which codecs have .vxd files? The ones I know of usually have .DLL and .ACM file extensions. It's generally bad practice to do something complicated like a full video codec in kernel-space, so a .vxd usually means support for hardware accelerated decoding. I recall for example that anything below a 100MHz Pentium was too slow to decode MPEG-1 video realtime (usually VideoCD), so computers that needed MPEG-1 playback used a dedicated decoder card in the system to offload the task. Is that what the codec was?

[watlersworld]

I was thinking of Software codecs. See vdk3211w.vxd and vdk32116.vxd here: http://stephan.win31.de/w31mm_en.htm

Some virtual device drivers are .386 and others .VXD. Does the extension make any difference?

The win16 hardware CODEC drivers (AC97 and HDA) do not require extra VXD support. I do not know about MPEG decoder cards or VIDEOCDs.

I was told that Win3x can run fewer than 255 EXE/DLL at once. Does the VXD have similar limits?

[joncampbell123]

No, i don't consider implementing a software codec in kernel space, yet apparently that's what Voxware has done. Okay...

386 and VxD are the same thing: a linear executable that Windows loads into the kernel.

AC97 and HDA hardware use memory-mapped IO and do not use I/O ports (as far as I understand). It's also very unlikely that DOS programs would want to poke at it. The way Win16 works, it's perfectly possible to poke at hardware MMIO registers, and again, interrupts not taken by VxDs are reflected to user-mode, so it makes sense that AC97 and HDA could be implemented entirely in user-space.

I've never heard of the 255 EXE/DLL limit, but I can only assume there's no VxD limit other than available memory. Since the VxDs are added in a linked list I can only think of possible performance degradation searching the list.

[watlersworld]

I see a myriad of files and folders. Could you give a simple *.bat file that sets paths and compiles a quick example?

Is just VDAG31WH.HLP required or do I need the entire DDK?

Are you planning some sort of run time code editing for *.VXD files under DOSBox-X or is it just your choice of emulators?

Are there any pascal-like calls?

[joncampbell123]

For this project, you will need the entire DDK, but mostly VDAG31WH.HLP and any include headers under 386/include to gather constants and structures.

The vxddef and vxdcalls are under windows/w9xvmm. Use ./make vxddef in that directory to regenerate the dev_vxddev*.h headers from the vxddef file.

An experimental build of a VxD that does nothing is in windrv/dosboxpi/win9x. The system is set up to use only DOSLIB tools, GCC 6.x, and Open Watcom to compile a VxD. You will need GCC 6.x or higher, target i386. A script is provided in the top level of DOSLIB called buildgcci386-71.sh that will build GCC+binutils binaries wherever it's executed. On my system, I built it under my home directory as follows:

• cd ~
• mkdir -p src
• cd src
• cp /usr/src/doslib/buildgcci386-71.sh ~/src/
• ./buildgcci386-71.sh

It will produce binaries under ~/src/gcc-7.1.0-i386-cc-build

[joncampbell123]

By design, my VxD development toolchain does everything possible to avoid using MASM or Microsoft's linker, and instead compile VxDs with only open source tools.

[joncampbell123]

I feel I need to clarify that all I am asking for is help adding all the VxD call definitions to the dev_vxddev.h headers in DOSLIB through the .vxddef files in the same directory, not yet to write a VxD.

[watlersworld]

OW 1.9 does WDEBUG without the MS DDK. Why do I need GCC to build a *.VXD?

[joncampbell123]

I recommend using Open Watcom v2.0 here on github rather than 1.9.

GCC (compiled to target the 386) is perfect for compiling C and assembly language for the flat 32-bit environment of the Windows kernel. GCC also has inline assembly language features that are needed to call into VxDs. I tried asking the Open Watcom developers for the same features, but they couldn't add them.

[watlersworld]

Do the Open Watcom V2.0 tools still work under win3x?

I never could get cygwin to work on properly on any windows version.

Do you think that languages such as CC386,Orange C,Pelles C or Xblite might be able to create VXD code?

[joncampbell123]

Open Watcom 2.0's gui still works under Windows 3.1.

The tools here in the DOSLIB are meant for use on Linux at the current time.

EDIT: Meaning that they're meant for use on Linux to cross-compile to DOS and Windows.

[watlersworld]

Good to hear that.

I always wondered how linux GCC does with win32s apps. Does it do a good job on the PE with RELOCs? Do you think you could you provide a linux gcc win32s GUI skeletal export that I could add as a code generator module of OOFD? http://turkeys4me.byethost4.com/oofd35.zip

Why port the entire DDK when you could just create autonomous and functional examples as OW did?

[joncampbell123]

GCC does perfectly fine with Win32, especially if you tell it that the target is i386-pe. It can also do win64. You can tie it together with MinGW headers and write native applications. Binutils is able to generate relocations, yes.

On Linux, I was able to make a MinGW-based gcc cross compiler to write Windows XP applications in C++ just fine using GCC 4.4 back in the day. As far as I know the only requirement for MinGW-based GCC compiling is the Microsoft C runtime DLL, which isn't on the system by default below Windows 95 OSR2. You can get a runtime DLL for it on the original Windows 95 install. I'm not really sure if MinGW can target Win32s (Windows 3.1), it's worth a try.

[watlersworld]

I know that Win3.x can use RSXNT to compile win32s applications, and that regular Mingw 3.2.3 and TDM-GCC 3.0.0 to 5.2.0-3 also create win32s programs.

Mingw(Mini GCC for Windows) was rather buggy during its infancy. I was told that since the beginning the full versions of GCC had better support for all targets.

Is making Win32s GUI programs a triviality for full Linux GCC versions?

[joncampbell123]

It's fairly trivial, yes. If you can write a Win32 app with MinGW in Windows and makefiles, you can re-use the same experience for cross-compiling in Linux to Win32.