Exploring hard disk size limits

[andrewbird]

I'm playing with hard disk sizes on FDPP, as my interest was piqued because of issue https://github.com/FDOS/format/issues/1. To test things I'm using a sparse image file, subsequently formatted using linux mkfs.fat as FAT32. I find that I can't quite get to 2 TiB filesystem size. I understand that we have to wrap the filesystem with an MBR/partition table, so the resultant disk size is slightly greater. As I increase the filesystem size towards the maximum theoretical size for FAT32 (2TiB - 512B) the error is less than graceful, boot fails as the disks partially get renumbered and ultimately command.com can't be found.

Results

(2TiB - 5*512B) : boots fine, all space seems to be available (2TiB - 4*512B) : fdpp falls back to CHS addressing, space limited to 8GiB, boot fails due to drive number confusion.

C: HD1, Pri[ 1], CHS=    0-1-1, start=     0 MB, size=    10 MB
Drive is too large to handle, using only 1st 8 GB
 drive 81 heads 255 sectors 63 , total=0x3b-00000001
can't use LBA partition without LBA support - part Pri:1 FS 0c start    0-1-1, end 65535-254-63
Drive is too large to handle, using only 1st 8 GB
 drive 81 heads 255 sectors 63 , total=0x3b-00000001
can't use LBA partition without LBA support - part Pri:1 FS 0c start    0-1-1, end 65535-254-63
D: HD3, Pri[ 1], CHS=    0-1-1, start=     0 MB, size=  2000 MB
E: HD4, Pri[ 1], CHS=    0-1-1, start=     0 MB, size=  2000 MB
F: HD5, Pri[ 1], CHS=    0-1-1, start=     0 MB, size=  2000 MB
Drive is too large to handle, using only 1st 8 GB
 drive 81 heads 255 sectors 63 , total=0x3b-00000001
Drive is too large to handle, using only 1st 8 GB
 drive 81 heads 255 sectors 63 , total=0x3b-00000001
can't use LBA partition without LBA support - part Pri:1 FS 0c start    0-1-1, end 65535-254-63
CONFIG.SYS error in line 11                                                    
>>>device=dosemu\umb.sys
          ^
UMBs unavailable!
CONFIG.SYS error in line 14
>>>devicehigh=dosemu\ems.sys
              ^
UMBs unavailable!
CONFIG.SYS error in line 16
>>>devicehigh=dosemu\emufs.sys /all
              ^
UMBs unavailable!
CONFIG.SYS error in line 17
>>>devicehigh=dosemu\cdrom.sys
              ^
Process 0 starting: F:\command.com /e:384 /p /k %FDPP_AUTOEXEC%
ERROR: Screen size is too large: 172x49, max is 160x100, Using 160x49
Bad or missing Command Interpreter: F:\command.com

The fall back to CHS addressing and geometry truncation to 8GiB seems to have been inherited from FreeDOS, which may have been fine when disks had physical geometry that could be relied upon, and now I'm wondering if there's a better way to handle this in Dosemu?

[stsp]

Does this truncation happen even with the redirector and your large disk extensions?

[andrewbird]

No, it's in FDPP's disk init stage so only affects block devices.

[andrewbird]

Although I don't know what would happen if I had a linux disks > 2TiB that dosemu would like to convert to fatfs.

[stsp]

So I guess it is supposed to work, because, no matter of truncation, there is a system area in the beginning of an FS. And dosemu is allocating the space sequentially, so the needed files are accessible. So I guess you only need to fix the numbering problems?

[andrewbird]

So I guess you only need to fix the numbering problems?

I suspect the drive is invalidated because of can't use LBA partition without LBA support, perhaps the geometry truncation doesn't need to flip the drive into using CHS accesses. On the other hand how safe is it to shorten a disk's geometry when an existing filesystem on there may already extend into a now inaccessible region? Perhaps dosemu should just ignore disks greater than 2TiB instead?

[stsp]

I am not actually sure. I think if you mean a single-partition hdimage, then it would be strange to ignore. No matter how large the drive is, or the partition is, the fatfs can't be that big itself. What exactly scenario is there that fatfs is partially inaccessible?

[andrewbird]

No matter how large the drive is, or the partition is, the fatfs can't be that big itself.

I agree if you are referring to dosemu's fatfs as that's only FAT16, but if you mean FAT in general then FAT32 can reach that size (just).

What exactly scenario is there that fatfs is partially inaccessible?

Disk containing one FAT32 partition of size (2TiB - 512B) is shared (either as dual boot with Windows or as removable drive moved between machines). On FDPP/FreeDOS that's seen as too big and geometry accessible is only 8GiB, but filesystem extends beyond that. I suppose that if DOS were to write a file that crossed the 8GiB boundary, it would only be able to write a portion of it.

[stsp]

But 8Gb is CHS-mode limit. If we don't fall back to CHS (which is a bug, right?), why do we care about 8Gb?

[andrewbird]

If we don't fall back to CHS (which is a bug, right?), why do we care about 8Gb?

Yes I think it's a bug to switch from LBA to CHS just because the disk is too big. It probably came about because early LBA implementations were seen as buggy and if any geometry oddities were encountered it was thought safer to use the CHS. Remembering of course that when LBA came out there were no multi-terabyte disks, and such large values were preposterous. So if we remain in LBA mode, presumably we would only have to truncate the geometry to be the maximum disk offset that FDPP can reach with 32bit variables?

[stsp]

I don't know the technical pit-falls. But I think if we can, we should support fat32 of any possible size. If that needs a 64bit variables - why can't we do that?

[andrewbird]

If that needs a 64bit variables - why can't we do that?

Not sure how to use 64bit variables in FDPP, I get confused about what's 16bit code and what isn't. For instance I needed to change a printf('%d') for a ULONG parameter earlier and I expected to use '%ul' but the compiler showed me that ULONG is actually 'unsigned int'.

In the ticket I linked earlier there was some discussion about needing 64bit math helpers for each compiler (Watcom / Gcc). Now I don't expect that's the case with FDPP as we don't use those 16bit compilers, but what is required to change a uint32_t to uint64_t?

[ecm-pushbx]

With MBR partitioning you only need "33-bit LBA" at most because your partition start / hidden sectors and partition / file system size are both limited to 32-bit each. There was a FreeDOS kernel patch by Tom Ehlert floating around that added GPT support however, in that case the lower-level (DOS drive to int 13h sectors layout) handling should support the full 64-bits.

[stsp]

ULONG is a 32bit type. Which is why it is "unsigned int" on 64bit platforms (and unsigned long on 16 or 32bit platforms). But you can still use uint64_t or long long.

[stsp]

(DOS drive to int 13h sectors layout) handling should support the full 64-bits.

I hope at least the LBA subset of it, already does.