BPB_HIDDEN always stores absolute LBA value

[boeckmann]

This is in contrast to MS-DOS and FreeDOS, where the hidden sectors are relative to the container (Disk or extended partition start).

EDR uses the BPB_HIDDEN internally to do the sector calculations in drbio\disk.asm. It should be considered to add a separate field in the UDSC data structure for this and use BPB_HIDDEN like in MS-DOS / FreeDOS.

[ecm-pushbx]

The BPB field for hidden sectors is chiefly for boot sector loaders. And they need it to be relative to the disk unit, not an extended partition. MS-DOS may not have expected to make logical partitions bootable, but if we do do that then the hidden sectors should be in the disk unit.

[ecm-pushbx]

This is in contrast to MS-DOS and FreeDOS, where the hidden sectors are relative to the container (Disk or extended partition start).

EDR uses the BPB_HIDDEN internally to do the sector calculations in drbio\disk.asm. It should be considered to add a separate field in the UDSC data structure for this and use BPB_HIDDEN like in MS-DOS / FreeDOS.

What field is used to find the extended partition start and thus the absolute address of the partition in those kernels? I don't see any such field in https://fd.lod.bz/rbil/interrup/dos_kernel/2f0803.html#table-02603

[ecm-pushbx]

This is in contrast to MS-DOS and FreeDOS, where the hidden sectors are relative to the container (Disk or extended partition start). EDR uses the BPB_HIDDEN internally to do the sector calculations in drbio\disk.asm. It should be considered to add a separate field in the UDSC data structure for this and use BPB_HIDDEN like in MS-DOS / FreeDOS.

What field is used to find the extended partition start and thus the absolute address of the partition in those kernels? I don't see any such field in https://fd.lod.bz/rbil/interrup/dos_kernel/2f0803.html#table-02603

https://github.com/FDOS/kernel/blob/8552d833bbca657591fa8e6f7276d307d7d0c7c6/hdr/device.h#L239 ddt_offset is behind the structure documented in the Interrupt List link. It is used as the absolute partition start: https://github.com/FDOS/kernel/blob/8552d833bbca657591fa8e6f7276d307d7d0c7c6/kernel/dsk.c#L837

[boeckmann]

The BPB field for hidden sectors is chiefly for boot sector loaders. And they need it to be relative to the disk unit, not an extended partition. MS-DOS may not have expected to make logical partitions bootable, but if we do do that then the hidden sectors should be in the disk unit.

Well then this would also be an issue for the FreeDOS kernel, which currently makes BPB_HIDDEN relative to the extended partition. The question is if it should be prioritized to stay compatible to MS-DOS or allow the boot from logical partitions. No matter how it is decided, it would be preferable for FreeDOS and EDR-DOS to do it the same.

ddt.offset is indeed the variable used by FreeDOS to do the determine the absolute sector addresses.

[ecm-pushbx]

MS-DOS v6.22 certainly doesn't have a field at the position of FreeDOS's ddt_offset. Its UPBs are 64h bytes in size. After running int 2Fh function 0803h the pointer arithmetic shows this:

-h word [ds:di] - di
0064  decimal: 100
-

[ecm-pushbx]

The UPB listing in the Interrupt List says:

---fixed media---
 47h    WORD    partition (FFFFh = primary, 0001h = extended)
        always 0001h for DOS 5+
 49h    WORD    absolute cylinder number of partition's start on physical drive
        (FFFFh if primary partition in DOS 4.x)

These fields are used in this way if the "fixed media" flag in the word at 23h is set. MS-DOS v6.22's block device does still support a value other than 1 in the word 47h, but a primary partition does actually have the value 1 there and then a value of 0 in the word 49h.

The word 49h is added to the Cylinder value of the CHS tuple -- after the tuple is calculated from the "in-extended partition" logical block address (sector number in partition plus hidden sectors). The word 49h holds the Cylinder value of the start of the extended partition that holds the logical partition. In a test case created using MS-DOS v6.22 fdisk, all hidden sectors of 3 partitions (one primary, two logical) are 63 but the word 49h is different for all three.

Now to test MS-DOS v7.10 ...

[ecm-pushbx]

I actually checked MS-DOS v7.00 first, which does not support FAT32 but does support LBA. If the primary partition type is 0Eh (FAT12/FAT16 LBA) or a partition is within an extended partition type 0Fh (Extended LBA) then LBA is used to access it. This is marked by bit 10 (flag 400h) in the flags word at 23h. (Which is also used by FreeDOS to mark to use LBA.)

If the LBA flag is set for MS-DOS v7.00, then the hidden sectors at dword 17h contain the start of the partition within the entire unit, not just within the extended partition. The instsect /G auto switch will also detect this hidden sectors value then. There is a check for the word at 47h being 1 and the word at 49h being 0. If word 47h != 1 then skip next part. If word 47h == 1 and word 49h == 0 then skip next part too. (The observed LBA logical partition had word 47h = 1 and word 49h = 0.) Otherwise add dword 17h again. I'm not sure if this code path is ever taken, it seems like it would add the hidden sectors twice.

[ecm-pushbx]

MS-DOS v7.10 functions largely like v7.00, the 400h flag is for LBA access and also correlates to the hidden sectors containing the int 13h unit total LBA sector number. The partition types apparently determine whether flag 400h is set. If it is clear then word 79h == 1 means to add word 7Bh to the Cylinder part of the CHS tuple. (Different offsets than in v6.22 and v7.00 due to the EBPBs.) Obviously if you fill the hidden sectors with the unit total LBA and word 7Bh = 0 then even in CHS mode the kernel will find the correct data position, and would work with extended partitions not on cylinder boundaries as well.

What's different is that each BPB in the UPB is a FAT32 EBPB, with the FAT32-specific parts zeroed out except for FSINFO sector and backup sector both filled with 0FFFFh. The EBPB does contain the 12 reserved bytes at its trail, unlike FreeDOS-with-FAT32. Also unlike FreeDOS-with-FAT32 the field of 6 reserved bytes behind one of the BPBs is dropped instead.

EDR-DOS also changes the UDSC (unit descriptor) layout yet again. https://hg.pushbx.org/ecm/edrdos/file/6497e3a1a0c7/drbio/udsc.equ#l35 contains an EBPB length of 41 (29h), like FreeDOS-with-FAT32 excluding the 12 reserved bytes. The trailing 6 bytes (7 bytes here due to a byte presumably not used by EDR-DOS) is also preserved, alike FreeDOS-with-FAT32 and unlike MS-DOS v7.10. EDR-DOS adds two new fields after the common structure, totalling 4 bytes. (I think that brings EDR-DOS to the same UDSC length as FreeDOS-with-FAT32, because FD's ddt_offset also takes up 4 bytes. Yes, both are 88h bytes. As opposed to 64h for MS-DOS v6.22/v7.00, 96h for MS-DOS v7.10, and 68h for FreeDOS-without-FAT32.)

I will have to study to see whether EDR-DOS and FreeDOS can use a compatible content in the partition fields to match the MS-DOS v7.xx content, ie a 1 and a 0 respectively. In that case the hidden sector could always be the total unit LBA like for MS-DOS v7 when LBA is enabled.

[boeckmann]

Regarding UDSC_INT13EXT I found no evidence that this is actually used except of setting it on drive enumeration. UDSC_INT13BITS is currently used as a one-bit flag to indicate, if ext INT13 is supported.

As a first step to MS-DOS compatibility, I think we should try to drop UDSC_INT13EXT and UDSC_INT13BITS altogether and set 0x400 of the flags field as LBA indicator, like MS-DOS. Question would be if this flag should also be set on 0x06 and / or 0x0b partition types (a.k.a do LBA on these partitions).

The rest I have to investigate anytime next week. I am currently short on free time. Nice detailed investigation, as always 👍

[boeckmann]

I removed UDSC_INT13EXT and UDSC_INT13BITS via 99765bd.

[boeckmann]

Todo (first educated guess):

• adjust UDSC_TIMER low and high word (corresponding to offset 47h and 49h of DOS 6.22 UPB but shifted because EBPBs)
• make UDSC having the same size of MS-DOS 7.1 UPB
• store BPB_HIDDEN relative if in CHS mode
• adjust read / write routine to respect relative BPB_HIDDEN

[ecm-pushbx]

I don't think that we should emulate the CHS mode hidden-in-extended-partition approach, as it also limits extended partitions to only work correctly on cylinder boundaries I believe. As I noted in the FreeDOS report if we set UDSC_TIMER to word 1, word 0 we would stay compatible to readers of the LBA- and FAT32-enabled UPB of MS-DOS v7.10. If the LBA flag is clear and UDSC_TIMER words are a 1 and a nonzero (which we wouldn't ever create ourselves) we could still support this case by multiplying the second word by the amount of sectors per cylinder, then adding that to the absolute sector number. But store 1 then 0 when we do the login ourselves.

[boeckmann]

This seems to contain a bug: https://github.com/SvarDOS/edrdos/blob/b328744eb2f86d597bda90029565acc0ac5e078b/drbio/disk.asm#L700-L702 Because UDSC_BPB_LENGTH is 41 and not 53. So the signature is not checked. I will adjust UDSC_BPB_LENGTH 53 and adjust the UDSC structure accordingly.

Edit: magic value 11 seems to be the BPB bootsector offset.

[ecm-pushbx]

This seems to contain a bug:

https://github.com/SvarDOS/edrdos/blob/b328744eb2f86d597bda90029565acc0ac5e078b/drbio/disk.asm#L700-L702

Because UDSC_BPB_LENGTH is 41 and not 53. So the signature is not checked. I will adjust UDSC_BPB_LENGTH 53 and adjust the UDSC structure accordingly.

Wait a moment, is this actually supposed to read a FAT32 EBPBN though? Shouldn't it rather be OLD_UDSC_BPB_LENGTH? I don't see this dispatching based on FAT12/FAT16 vs FAT32. Because it is only used on diskettes and the CHS = 0:0:1 is hardcoded I think it should be handled as FAT12/FAT16.

Edit: magic value 11 seems to be the BPB bootsector offset.

And the magic 2 is bpbnExtBPBSignature within the BPB New fields.

[boeckmann]

Wait a moment, is this actually supposed to read a FAT32 EBPBN though? Shouldn't it rather be OLD_UDSC_BPB_LENGTH? I don't see this dispatching based on FAT12/FAT16 vs FAT32. Because it is only used on diskettes and the CHS = 0:0:1 is hardcoded I think it should be handled as FAT12/FAT16.

May it be to support some other removable media other than diskettes and possibly a larger size? Like a ZIP disc or something?

[ecm-pushbx]

https://github.com/SvarDOS/edrdos/commit/162ce5b9fadd8b76f093e9b7afec372040eff34d#r139019375 comment added. I don't think this change is entirely correct.

[ecm-pushbx]

Wait a moment, is this actually supposed to read a FAT32 EBPBN though? Shouldn't it rather be OLD_UDSC_BPB_LENGTH? I don't see this dispatching based on FAT12/FAT16 vs FAT32. Because it is only used on diskettes and the CHS = 0:0:1 is hardcoded I think it should be handled as FAT12/FAT16.

May it be to support some other removable media other than diskettes and possibly a larger size? Like a ZIP disc or something?

Don't think so. You could add the FAT32 dispatching if you really want it to support that, but unconditionally reading the EBPBN seems wrong.

[boeckmann]

Wait a moment, is this actually supposed to read a FAT32 EBPBN though? Shouldn't it rather be OLD_UDSC_BPB_LENGTH? I don't see this dispatching based on FAT12/FAT16 vs FAT32. Because it is only used on diskettes and the CHS = 0:0:1 is hardcoded I think it should be handled as FAT12/FAT16.

May it be to support some other removable media other than diskettes and possibly a larger size? Like a ZIP disc or something?

Don't think so. You could add the FAT32 dispatching if you really want it to support that, but unconditionally reading the EBPBN seems wrong.

The longer I look at this dd_medchk the more suspicious this looks. Seems the intention is to compare the volume serial as disk changed indicator, but it does not try to read the FAT-16 volume serial at all. It instead looks for signatee 29h as indicator for a FAT-32 volume serial, but does not test properly for FAT-32 by comparing root dir entries to zero?!?

[boeckmann]

check for FAT32 is now added to dd medchk via e1d3461.

[ecm-pushbx]

I changed my MS-DOS fork to also use the hidden sectors (dword [17h] in the UPB) for the unit-absolute start of the partition, leaving 0001h/0000h in the IsMini and Hidden_Trks fields.

I also changed it to use only the LBA values in the partition tables, and never read the CHS values.

I also added LBA detection and using LBA functions.

[boeckmann]

Nice! Does the implementation of LBA unlock the use of larger partitions, or must other areas of the DOS 4 code be modified to achieve this?

While skimming over DOS 4 code I noticed they are speaking of "mini partitions". Never heard that term before. I assume this is what we know today as logical partition?

Do you have further plans on improving the MS-DOS 4 code?

[ecm-pushbx]

Nice! Does the implementation of LBA unlock the use of larger partitions, or must other areas of the DOS 4 code be modified to achieve this?

It appears that the usual FAT16 limit (a bit below 64 Ki clusters 64 sectors per cluster 512 bytes per sector = nearly 2 GiB) is in effect, same as for MS-DOS v7.00. At least a cursory test of maximum size FAT16 partitions created by MS-DOS v7.00 fdisk seems to indicate they work.

While skimming over DOS 4 code I noticed they are speaking of "mini partitions". Never heard that term before. I assume this is what we know today as logical partition?

Yes, mini disks are logical and/or extended partitions. As yet the code accepts one primary partition for each of one or two hard disks and a chain of mini disks for (possibly more?) hard disks. (I only tested with two hard disks so far.) I think the code expects that per MBR there is only one extended partition and per EPBR there is at most one logical partition and at most one further (nested) extended partition.

Do you have further plans on improving the MS-DOS 4 code?

Indeed. Just today I created a 20-odd GiB disk image (on my local desktop to not strain the disk space use on our server) and partitioned and formatted it with MS-DOS v7.00. This was intended as a test of my LBA support. That seems to work fine with yesterday's build. However, I found that I misunderstood the base structure of nested EPBRs so my change to calculate EPBR offsets only using the start LBA fields failed for more than 2 logical partitions.

I fixed this, of course: https://hg.pushbx.org/ecm/msdos4/rev/d56368bd703e but not before I added a fix to my CHS tool Extension for lDebug to correctly display the EPBR start addresses in the chstool walk 81 extended mode: https://hg.pushbx.org/ecm/scanptab/rev/522be373bef1

Other than that obviously I want to work more on the kernel, maybe some on the shell too. A priority is getting the programs to work on other kernels as well, eg to eliminate the "Parse Error 3" on invalid switches. VER /R and MEM /P would be good to have. UMA and HMA use would be nice but probably not a priority. FAT32 and FAT+ support look like a lot of work. LFN "awareness" (skipping LFNs rather than matching them as labels) is probably easy enough. Writing of the partition walking it would probably be desirable to support multiple primary partitions and more hard disks, as well as multiple extended partitions per MBR/EPBR. That implies an option for DLA sorting in FreeDOS style vs later MS-DOS style.

As usual, I am zoomed in on the kernel as my primary project. I will likely add the EMS commands from the release's Debug to my MSDebug fork, but not all the changes to the string display architecture. (They do make it easier to localise the application but that also isn't a priority for me.)

[boeckmann]

FAT32 would be very much work (the effort put into the EDR kernel by Udo proves that). UMA and HMA probably too, at least for the brain. Now that Microsoft basically out of nowhere released DOS 4, there is a chance that DOS 5 will eventually be released too. So to me it is questionable if it is worth to put so much effort into theUMA HMA thing, or instead bet on the release of DOS 5/6. But its hobby, so fun mainly dictates what we do :)

On the net people repeat saying that DOS 4 was particularly a bad release because of its memory consumption compared to DOS 3.3. I was not in existance at that time, but I am wondering if this can be fixed without compromising the functionality, so basically making a better 3.3 for ancient PCs.

[dptpirate]

Assuming there is no incompatibility with software that does run on DOS 3.x. I don´t see why DOS 4.x would be a bad choice for an XT if you can max it out at 640k of RAM. I have an Amstrad PC1512 and I can´t remember any software that does not run with 512k of RAM under DOS 3.x. If you max it out at 640k RAM and run DOS 4.x, you still have more than 512k RAM that is free. I think the complaints with DOS 4.x were mostly with people that had 286/386 PCs.