FORMATting a phantom B: diskette leads to SvarCOM reload failure

[mateuszviste]

reported by Willy Spiegl: apparently he boots his system from a SvarDOS floppy and then tries to format the floppy. He complained that afterwards COMMAND.COM fails to reload.

It's two cases, really:

1. FORMAT A: leading to SvarCOM being unable to respawn is normal and expected, since it was booted from A: in the first place. Once FORMAT is done with its job, SvarCOM's RMOD will try to reload A:\COMMAND.COM, but this file vanished so the system is bricked. Nothing to fix there.

2. FORMAT B: is a slightly different story. I assume this is a single-diskette system, so drive B: is most probably a phantom drive. In such configuration I'd expect DOS to print out its "insert a diskette in drive B:...", and then, once the job is done, ask for a diskette in A: again. But here it does not happen:

Not sure who is to blame in this yet... EDR? FORMAT? SvarCOM?

[boeckmann]

He also reported this to me. Could be anything of the three components mentioned. We should also test if the single drive copy from a: to b: etc. works. If I am right, one should be able to do a copy a:\test.txt b:\test.txt even if there is only a single floppy drive installed. SvarDOS (kernel?!?) should then ask for the disks to be swapped...

[boeckmann]

I tested a few things:

formatting b: in a DUAL drive system works under SvarDOS 20240915 floppy disk and under EDR only disk (kernel + command.com):

formatting b: in a SINGLE drive system fails for me in 86box. Not sure this is something that should be expected to work at all? Probably not?

copy a:\config.sys b: works for a single drive system. Kernel then requests to change disks several times.

[mateuszviste]

formatting b: in a SINGLE drive system fails for me in 86box. Not sure this is something that should be expected to work at all? Probably not?

Dunno, but intuitively I'd expect a similar behavior as with COPY... perhaps FORMAT needs to tell DOS that it is about to access B:, so DOS can do the ghostly magic. I plan to test it with MSDOS soon.

[boeckmann]

Does the ERR 2, LOAD FAILED come from the SvarCOM "loader TSR"? Perhaps this could be placed inside a "press key to retry..." loop? In case someone runs it from a: and the wrong disk is in the drive...

[mateuszviste]

Does the ERR 2, LOAD FAILED come from the SvarCOM "loader TSR"? Perhaps this could be placed inside a "press key to retry..." loop? In case someone runs it from a: and the wrong disk is in the drive...

Yes, it comes from RMOD, ie. SvarCOM's Resident MODule. It is already in a "retry after keypress" block, although the message is not very explicit about it because I wanted to keep the whole thing as tiny as possible and save every possible byte. But it would be nice indeed to have it somewhat clearer, even if it costs 10 bytes or so of resident footprint.

[mateuszviste]

I confirm this works fine on MS-DOS. Tested under VirtualBox with MS-DOS 5.0.

I booted from A: on a single-diskette system, typed FORMAT B:, got asked to insert a diskette into B: so I ejected the diskette with the system and inserted a different floppy, then FORMAT formatted it and asked to place a diskette into A: - at this time I inserted my original (system) diskette, and after pressing a key I was happily back at the prompt.

Question now is why this does not work with SvarDOS. My primary suspect is FD-FORMAT, which is probably supposed to access B: through DOS, so the kernel can do the phantom-masking-swapping thing, and undo it when the application stops and RMOD tries to access A: again.

[boeckmann]

Can you try it with FD format under the MS-DOS kernel, so we can be sure? If this works we have to look at the EDR code.

[mateuszviste]

Can you try it with FD format under the MS-DOS kernel, so we can be sure? If this works we have to look at the EDR code.

Yes, that was my plan. Using FD FORMAT on MS-DOS fails, just like it was failing with SvarDOS for @fritzmueller. It definitely seems to be a FD FORMAT bug. Also, FD FORMAT fails to format the (virtual) floppy at all. So it's two bugs, actually.

Step 1: (FD FORMAT)

When FD FORMAT asks to insert a new floppy, I inserted a different floppy image. FD FORMAT fails to format it with some cryptic code, and returns to prompt without asking to insert a diskette into drive A:

Step 2: (MS FORMAT)

I have inserted back my MS DOS 5.0 system diskette and re-did the test with MS FORMAT. When asked to insert a diskette I inserted the same image as I submitted to FD FORMAT, but MS FORMAT managed to format it without any issue. Once done, it asked to insert the system diskette back.

[mateuszviste]

Another test: I used FORMAT.COM from MS-DOS 6.0 with SvarDOS. It works perfectly!

At this point I am really tempted to drop FD FORMAT and replace it by MS FORMAT from the MIT-released MS-DOS 4.0. This code would need to be made more tolerant vis-a-vis the DOS version (expects "4.0" while EDR presents "6.0") and some work would be needed to translate messages and make it build with NASM, but it appears to be a more robust solution, and more compact at the same time. And with a nicer license. The one drawback is missing FAT32 support. Perhaps this wouldn't be too hard to add, if it is only about generating an empty boot sector structure. Or keep FD FORMAT around just for the sake of FAT32.

[boeckmann]

What I do not like about FD format is how it develops into a really bloated thing. Jeremy had to change the memory model to compact when including the translation library, bloating it even more. I guess the most recent build will be significantly larger than the one that is currently shipped by SvarDOS.

On the other side, MS format has no idea about FAT-32 at all, like you said. So not sure what is worse. Probably not having FAT-32 support at all...

[ecm-pushbx]

I wanted to do lDOS's MS-DOS v4 kernel next but I could try my hand at its FORMAT tool first if it is desired to convert it to build with NASM + WarpLink.

[boeckmann]

Question is what would be easier. Implanting FAT-32 supprt into the MS format or doing a clean sheet reimplementation and using MS format as a reference for implementing the nasty details. Probably extending MS format would be less effort, I guess, but not neccessarly easier...

[mateuszviste]

I wanted to do lDOS's MS-DOS v4 kernel next but I could try my hand at its FORMAT tool first if it is desired to convert it to build with NASM + WarpLink.

Having a "legacy-powered" FORMAT available would be really nice, esp. given the current (buggy, bloated, viral license...) solution we borrowed from FreeDOS. But the missing FAT-32 is really a mood killer. Do you have any idea if it would be realistic to add FAT-32 into MS FORMAT 4.0? My naive assumption is that all the low level hardware-related magic is the same, but MS FORMAT needs to know how to fill a FAT32 boot sector. But I know nothing, so it's just words :) I also don't know how good is the LBA support in MS FORMAT (if any), and whether or not it has some hard limits on the size of supported drives...

[ecm-pushbx]

I wanted to do lDOS's MS-DOS v4 kernel next but I could try my hand at its FORMAT tool first if it is desired to convert it to build with NASM + WarpLink.

Having a "legacy-powered" FORMAT available would be really nice, esp. given the current (buggy, bloated, viral license...) solution we borrowed from FreeDOS. But the missing FAT-32 is really a mood killer. Do you have any idea if it would be realistic to add FAT-32 into MS FORMAT 4.0? My naive assumption is that all the low level hardware-related magic is the same, but MS FORMAT needs to know how to fill a FAT32 boot sector. But I know nothing, so it's just words :)

I do think that it shouldn't be very difficult, much easier than supporting FAT32 in the kernel / file system driver.

I also don't know how good is the LBA support in MS FORMAT (if any), and whether or not it has some hard limits on the size of supported drives...

Limits may exist but are probably not that difficult to fix. I will have to study how FORMAT accesses drives, it may be that LBA support doesn't even need any changes to the application, just a kernel that supports LBA.

[ecm-pushbx]

https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/CMD/FORMAT/MAKEFILE#l54 Anyone have any idea what "convert" does?

[boeckmann]

https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/CMD/FORMAT/MAKEFILE#l54 Anyone have any idea what "convert" does?

Idea would be that it is something like EXE2BIN, so converting the .EXE to a .COM? But why don't they use EXE2BIN?

[ecm-pushbx]

The converted .com file is actually slightly larger:

~/proj/msdos4.hg/hg$ ls -l src/CMD/FORMAT/format.{exe,com} -gG
-rw-r--r-- 1 22875 Sep 19 11:08 src/CMD/FORMAT/format.com
-rw-r--r-- 1 22736 Sep 19 11:08 src/CMD/FORMAT/format.exe

The .com file is loaded as a flat-format .COM executable but at seek offset 16 it includes the original MZ header, and apparently all the relocations are still present as well.

[mateuszviste]

I do think that it shouldn't be very difficult, much easier than supporting FAT32 in the kernel / file system driver.

That is my general feeling, since FORMAT does not need to access the content of the drive, just initialize it. The only exception is FORMAT /S, but maybe it could simply call SYS instead of doing the work itself. Or maybe FORMAT/S could be removed.

Limits may exist but are probably not that difficult to fix.

When you say it, it sounds like an easy ride. :-) If you feel like investigating a little bit, I'd be very curious of your findings.

On another note, I wonder if there are other tools that it would make sense for us to pull from MS... thinking about ATTRIB, CHKDSK, FC, KEYB, LABEL, SORT, MEM... They are all very compact and MIT-licensed, but each need to be adapted to NASM + some assembly-level NLS solution would be needed (an ASM equivalent of SvarLANG)

Idea would be that it is something like EXE2BIN, so converting the .EXE to a .COM?

The FORMAT.COM binary has actually an MZ header hidden after 16 bytes of "something":

[ecm-pushbx]

Ah, the only difference is the added 16-byte header and a 123-byte trailer. If I cut out the file using tail --bytes +17 then this binary exactly matches the .exe.

[ecm-pushbx]

The trailer seems to just process the MZ header and transfer control to the relocated image. Talk about useless.

[boeckmann]

The trailer seems to just process the MZ header and transfer control to the relocated image. Talk about useless.

Any idea why they do this? Even DOS 1 has the ability to do MZ relocations?

[mateuszviste]

Any idea why they do this? Even DOS 1 has the ability to do MZ relocations?

Maybe the idea is to avoid the user to executing some 3rd-party FORMAT.EXE while thinking it's the normal DOS format.

[ecm-pushbx]

The trailer seems to just process the MZ header and transfer control to the relocated image. Talk about useless.

Any idea why they do this? Even DOS 1 has the ability to do MZ relocations?

I don't think all DOS v1 releases support MZ executables. But this is moot since the application checks for the int 21.30 version anyway and won't start if it isn't 4.00. (Circumventing this with callver 4.00 brings up a different error running on FreeDOS in dosemu2.)

[mateuszviste]

According to the changelog, FORMAT became an EXE file in v2.28. So perhaps they kept the COM extension in case someone, somewhere relied on invoking "FORMAT.COM".

In any case, it's not a concern for us.

[boeckmann]

Is there anything in the source that prevents it from actually being converted into a .COM file? So they had to use this "trick" to pretend it is a .COM file?

[ecm-pushbx]

Is there anything in the source that prevents it from actually being converted into a .COM file? So they had to use this "trick" to pretend it is a .COM file?

It has relocations I think, so it cannot simply be loaded as a flat-format file without the help of this trailer. However, DOS is fully able to load a file named anything.com when it actually starts with an MZ header and will process the relocations then.

[ecm-pushbx]

Hmm, seems that the stack is initialised with a zero word on top instead of the last part of the "-Stack!-" pattern. Could be a reason for converting, as this is not done for MZ headered files usually.

[ecm-pushbx]

No, the CS is different from the PSP and this word doesn't seem to be above ss:sp when control is passed to the relocated image.

[mateuszviste]

I could try my hand at its FORMAT tool first if it is desired to convert it to build with NASM + WarpLink

I did not know about WarpLink. Or - I did not know it's a useful thing nowadays. What version do you use? Is it the historical 2.7 build released into public domain in 1999? If it's useful (at least to you) I'll add it to SvarDOS packages.

[ecm-pushbx]

I could try my hand at its FORMAT tool first if it is desired to convert it to build with NASM + WarpLink

I did not know about WarpLink. Or - I did not know it's a useful thing nowadays. What version do you use? Is it the historical 2.7 build released into public domain in 1999? If it's useful (at least to you) I'll add it to SvarDOS packages.

Yes, I'm using the public domain build with one bugfix, and itself ported to be assembled by NASM (and linked by itself). Repo is at https://hg.pushbx.org/ecm/warplink/ and https://pushbx.org/ecm/download/old/warplink/ has my builds so far.

[ecm-pushbx]

To expand on the usefulness, it's an OMF linker that is in the public domain and (now) builds with only free software.

It is usually compatible with Microsoft's linkers, needing slightly different command line / response file contents but producing largely the same binary output.

Finally, its extended /mx map file output is useful for my convlist script and for use with ident86, moreso than the Microsoft linker map files that I've seen.

[ecm-pushbx]

The only different output I've seen is WarpLink linked files being shorter than others, with the remaining data in the longer file being all-zeroes padding.

[mateuszviste]

Yes, I'm using the public domain build with one bugfix, and itself ported to be assembled by NASM (and linked by itself). Repo is at https://hg.pushbx.org/ecm/warplink/ and https://pushbx.org/ecm/download/old/warplink/ has my builds so far.

Very cool! So far I have added the historical 2.7 version from 1999 into SvarDOS packages. I'd like to add your improved build as well, but the version is confusing - the binary says "release 0 by ecm". Any chance you could make it display some actual version? The same as the build would be perfect (like "20240801"), this way it would be considered by the SvarDOS package manager as an upgrade over the historical v2.7.

[mateuszviste]

PS. I can also provide you with svn access so you could upload updates of your softwares yourself if you'd be willing to, just tell

[ecm-pushbx]

Very cool! So far I have added the historical 2.7 version from 1999 into SvarDOS packages. I'd like to add your improved build as well, but the version is confusing - the binary says "release 0 by ecm". Any chance you could make it display some actual version?

Release 0 is the version. If/when I get around to updating WarpLink more the next version will be release 1, and so on.

The same as the build would be perfect (like "20240801"), this way it would be considered by the SvarDOS package manager as an upgrade over the historical v2.7.

I don't want to use the date as a version number.

PS. I can also provide you with svn access so you could upload updates of your softwares yourself if you'd be willing to, just tell

I prefer things being reviewed by others actually. And I barely know how to operate subversion.

[ecm-pushbx]

The message handling services are fairly complex. Besides, there is no /? help yet. And the error messages are not good even at best (running on its own kernel + command.com), and worse otherwise (running on FreeDOS with MS-DOS v4 command.com or FreeDOS + FreeCOM).

[ecm-pushbx]

Some of the sources use differing macros like https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/INC/MSGSERV.ASM#l119 where structures are given with some member fields initialised and others left as ? which my tooling doesn't support yet.

[ecm-pushbx]

Haven't figured out entirely what these M_RT macros are for: https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/INC/MSGSERV.ASM#l481

[ecm-pushbx]

https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/CMD/FORMAT/DISPLAY.ASM#l59 This includes sysmsg.inc which defines a bunch of macros.

Notably these to include further content: https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/INC/SYSMSG.INC#l274

Some of the macros will lead to including msgserv.asm which defines actual data or code: https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/INC/MSGSERV.ASM

The M_RT label is a structure instance, and for resident command.com it seems to be defined twice (in the same translation unit)? https://hg.pushbx.org/ecm/msdos4/file/756d583bcddb/src/INC/MSGSERV.ASM#l190

[mateuszviste]

This issue is worse than just SvarCOM being unable to reload itself after FD FORMAT. It is actually impossible to format a floppy on a diskless, single-floppy SvarDOS system. Very annoying.

(E)DR-DOS also came with FORMAT. Presumably with FAT32 support already. Maybe we could steal that?

[ecm-pushbx]

(E)DR-DOS also came with FORMAT. Presumably with FAT32 support already. Maybe we could steal that?

EDR-DOS format was just a patched FreeDOS format, eg to add support for 256 sectors/cluster.

[boeckmann]

EDR-DOS format was just a patched FreeDOS format, eg to add support for 256 sectors/cluster.

What exact version are we talking about? Is it anywhere available as source?

[boeckmann]

Skimming over the source it seems that FreeDOS format accesses the floppy via int 13 to do certain operations. But I see no check if drive B: is actually a real drive. I guess these calls must be remapped to drive A: to do the actual work on a single floppy system. How to determine if it is a single floppy drive? Via BIOS equipment info or better via some DOS call?

[ecm-pushbx]

https://fd.lod.bz/rbil/interrup/dos_kernel/21440e.html for the phantom drive detection I believe.

https://pushbx.org/ecm/download/edrdos/freedos/ for original DR-DOS Enhancement Project files.

[mateuszviste]

but maybe this phantom detection is not needed in the first place, if FORMAT would politely ask DOS for doing writes (int 26h), instead of hitting the BIOS? This might actually solve the missing "switch diskette" bug.

[mateuszviste]

about the phantom check -- this is how I do it in ethflop: https://sourceforge.net/p/ethflop/code/HEAD/tree/ethflop-client/trunk/ethflop.asm#l515

but I still think it is not FORMAT's job to detect this. IMO FORMAT should talk to DOS, so the kernel can figure out whatever ghostly logic is needed and do the right thing (translating B: to the proper BIOS id and asking user to switch floppies when needed).

[boeckmann]

I agree. However, in the code there is a special case in that sector 0 is read explicitly via INT 13:

https://github.com/FDOS/format/blob/8071c208afe445cdd000b456bab8625814a999a8/driveio.c#L415

But much of FloppyBootReadWrite is commented out, and it is not clear to me why they did not use INT 25 for it.

[boeckmann]

In floppy.c there is a ~600 line Set_Floppy_Media_Type function, which is basically riddled with INT 13 calls. Not sure if these can be simply left out. Also, INT 13 is used to format floppy disk tracks.

[mateuszviste]

as much as I can tell the MS-DOS format does not have the need to call the int 13h BIOS:

[boeckmann]

Format_Floppy_Cylinder uses an INT 13, but there is a DOS generic IOCTL (42h), which by the name does the same, and makes sure the right disk is in the drive. So this could perhaps be replaced. One INT13 call less.

https://github.com/SvarDOS/edrdos/blob/529e535fc260b8e2ffb1d1f5f12efb735ce5d24c/drbio/disk.asm#L1788