Extending dosbox.

[mateli]

I have been thinking about various ways to extend DosBox and make it faster. Most ideas I have is only useful with new/recompiled DOS applications. On the top of my list is the idea of having a way to load dynamically linked modules that are either dll-like files within the VM fully compatible with any DOS environment or maps into a DosBox extension. This idea resurfaced when working with Python which is a very slow language yet can do some tasks at great speed due to excellent native modules.

For example a DOS application that needs to use Zlib would link to a zlib.dll and if the application is not running in dosbox then that would be an x86 library. However if it is running in DOSBox then there will be a zlib module compiled for the host platform that runs directly on host hardware, however trough a dosemu library that among other things map dos paths into physical filesystem paths as well as allowing the module to "DMA" data directly into the guests memory.

Note that I have not specified how these things would work. One idea I have is to do this the way Win3mu works using the NE/PE file types for executables however while not implementing the Windows API at least not in it's fullest. Some Windows API:s could be very useful to have in dosbox itself such. For example threading and processes could be implemented inside dosbox where the interpreter simply has multiple contexts to switch between which should be way faster than implementing running x86 code inside the emulator. For starters this could be a developed as a drop in replacement for the HX DOS Extender but obviously easily be extended to with a much more extensive API trough native modules. If we use Windows files I would personally prefer that both 16 and 32 bit exe files are supported leaving us room to potentially implement the entire Win16 and Win32 API however this is not something that I would consider a priority.

Another way to do this could be to map instructions and/or interrupts to native modules. I am not really that interested in how this is done as long as there is a decently fast mechanism to call into these extension modules.

Once there are such an extension mechanism then I can easily use it to implement extensions like the following:

• Native zlib utilizing hardware compression on some platforms.
• Libarchive for more extensive archive file format support
• SQLite is probably much faster when running outside of the VM.
• vFS - accessing files more directly than going trough the DOS emulation. The vfs would map to the filesystem the same way that the DOS emulation does but also leave room for other interesting features such including memory mapping, "DMA"-ing data directly to/from DOS memory, access to network filesystems and having a mechanism for file locking. Without the module an application would fall back to using the DOS filesystem API. Some of these mechanisms such as file locking cannot be implemented with files that are also exposed trough he DOS API. If Implementing the Windows API the vFS can be designed to map into that API and for example solve the issue with the missing share.exe.
• WinSock/Sockets. If we have a HX DOS:ish API I probably implement something that is compatible with WinSock and the various Windows applications that depend on WinSock. Otherwise I might go with something more unixy. However the point being the creation of a network layer where the heavy lifting is done in native code. Such as handling packets and running CRC calculations on them.
• vNet. A monkier for various high level protocols including but not limited to HTTP, FTP, SFTP, SMTP etc. Leaving most of the processing to native code. For protocols that transfer files those can be downloaded directly by native code into the virtual filesystem while notifying the guest of progress and completion, appearing as an abortable background transfer.
• v2D. An api for rendering 2D graphics that is very abstract and therefore easy to port to different hardware. I do not plan on creating modules for low level graphics such as Allegro or SDL2 as I think that it is a bad idea to work that close to the hardware. Currently we have the problem that DOS applications are locked to a very narrow set of video hardware and the cost of emulating it is high. Creating libraries for Allegro or SDL2 would lock us into contemporary hardware that may be obsolete in 30 years and we would have the very same problem.
• v3D. Well DB2D plus one dimension
• vAudio. An abstract Audio library that includes 3D audio yet on some platforms only a subset of features may be supported.
• vMedia. Plays various media format both by streaming and by pointing vMedia to a file to play meaning no actual data has to go into the emulator or out from it.

Well the list can go on but I think the point is rather clear. Not only would such a framework enable DOS applications that support it to run a lot faster. It would also allow us to create libraries for DOS applications that do not need to be made inside DOS but instead can be written in whatever native compiler the host support, including C and native assembly.

One issue to think about here is how to handle multi-tasking on modules that may have to deal with concurrency issues. Running each task on a vCPU that each has an ID and communicating that ID to the module would probably be best.

[joncampbell123]

On a related note, I've thought about my work with building code into DOSBox-X to "link on demand" the BIOS and DOS interrupt handlers and how it might be used to "link on demand" DLL files that could override the built-in Windows 3.1 DLLs to accelerate things like VFW codecs or certain libraries. The NE format is weird but not too complicated to implement considering how poorly documented it is. It's a long term idea, probably not going to happen very soon.

[mateli]

While extending or emulating windows 3.x is interesting and has a reference implementation in win3mu it's not really want I'm after. Instead I'm interested in a windows-like dos extension for new applications. I'm a developer and I want to be able to target DOS with new applications.

Something similar to what win3mu does would fit my purposes but one problem however I don't need any Windows compatibility at all. Furthermore I would consider other binary formats. What I desire is to be able to run 16 bit code while have access to extend memory. The NE format combined with basic Windows compatibility is useful because of the available development tools. Such as for example turbo Pascal and perhaps Delphi as well as a decent amount of C/C++ compilers. However I don't want to go down the path of ReactOS or Wine that are only focused on compatibility.

The goal is to have a virtual machine that will run very well on modern hardware such as embedded ARM systems where DOSBox are already a lot more used than for example the JVM especially when it comes to games. The execution speed of x86 code within the emulator will hardly matter att all if most execution time is spent in native modules. Whats far more important is to keep x86 code compact which is why i prefer to have 16 bit executables over 32 bit.

Originally i mostly thought about extended instruction sets that can be wrapped into x86 assembly code. However I soon realized that this is bloated from the perspective of minimizing x86 code. Compared with the idea of having a standardized way for x86 code to call into native binaries. From a Virtual Machine perspective this isn't original. Java has its awful Java Native Interface. Python had modules which to me seems easier. I understand why a simple binding layer that in many cases can just take header files and generate for example JNI code is appealing. However in practice it's been rather difficult to create and maintain such libraries. Python Modules just assumes that wrapping a native library is going to take manual work and has an easy way to deal with that. And that approach resulted in an unprecedented amount of native libraries which is one of the things making python the top VM and language.

I however believe that DOSBox can easily grab that position because it already runs on a huge amount of hardware and have lots of applications. All it needs is to grow out of the emulator segment into the VM segment as a platform for NEW dos development.

[ThomasKjoernes]

What would be really cool with be to have a plugin system. Where you could install handlers for INTs, physical memory ranges and port I/O (read and/or writes).

[mateli]

As well as extensions to the instruction set. That would allow for very modular development.

[Torinde]

On a related note, I've thought about my work with building code into DOSBox-X to "link on demand" the BIOS and DOS interrupt handlers and how it might be used to "link on demand" DLL files that could override the built-in Windows 3.1 DLLs to accelerate things like VFW codecs or certain libraries. The NE format is weird but not too complicated to implement considering how poorly documented it is. It's a long term idea, probably not going to happen very soon.

That reminds me of:

Speaking of fragmentation, I've had this crazy idea for a long while that might be an interesting project. What you do with the project, is install the base OS (Windows 3.1, 95, 98, ME) and then overwrite system files with ones compiled from the project. You would have a project that, bit by bit, replaces the Microsoft components with open source ones, until eventually the open source bits can stand on their own as a complete Windows 3.1/9x/ME compatible system. Then once you've done that you can adapt the kernel to run on modern processors and hardware, and retro gamers could rejoice and enjoy their own games again without worrying where Microsoft would take you today :) It would be the 3.1/9x/ME complement to ReactOS's focus on NT-based Windows. Bonus points if the open source Windows clone runs atop FreeDOS.

Originally posted by @joncampbell123 in https://github.com/joncampbell123/dosbox-x/issues/188#issuecomment-219332908

Surprisingly close to Tennessene/jDOSBox/issues/8:

I ported DosBox and then added some more stuff on top of it. At one point it could even run a couple of Windows games without an OS (my attempt at Wine from scratch).

@Tennessene, maybe you can link James and Jon...

[Torinde]

I ported DosBox and then added some more stuff on top of it. At one point it could even run a couple of Windows games without an OS

Could that be a similar mechanism to the one used in PDOS University challenge x64?

a little glue (a loader that resolves msvcrt.dll references to the internal C library being used by bootx64.efi) and hey presto - a (subset) clone of Windows 11. Available here. ... There is also a hack that allows you to run some win32 programs instead. Just type "msdemo32.exe".