search

joncampbell123 / dosbox-x

DOSBox-X fork of the DOSBox project
GNU General Public License v2.0
2.81k stars 383 forks source link

Optimal GUEST hdd.img FAT Partition Offset and Cluster Size for HOST 4k aligned Block Device #5282

Open the-burrito-triangle opened 21 hours ago

the-burrito-triangle commented 21 hours ago

Question

CONTEXT: The Future is Now

Modern consumer storage media is generally aligned to 4KiB "physically" (4kn / 512e) for SSDs and HDDs. Since Windows Vista, most partitions start at 1MiB or sector 2048 on 512byte sector devices and end at a multiple of this offset to keep alignment. Depending on the host's filesystem, we can have logical allocation units >=4KiB. NTFS defaults to 4KiB cluster size. On Linux, BTRFS defaults to 16KiB "nodesize" for metadata block allocation and 4KiB "sectorsize" for data block allocation.

PROBLEMS: VM Storage is Like an Onion

A Guest virtual block device (stored as a file on a Host's file system) can potentially lead to Host-side write-amplification and reduced I/O performance due to a number of factors:

|-----------------------------------------------------------------------------------------|
| GUEST | File System  | logical allocation unit (FAT12/16/32 cluster size 512B to 32KiB) |
|  OS   | Block Device | logical sector size (512B: emulated 512n block device)           |
|-----------------------------------------------------------------------------------------|
| DOSBox-X HYPERVISOR                                                                     |
|-----------------------------------------------------------------------------------------|
| HOST  | File System  | logical allocation unit (e.g., NTFS default cluster size 4KiB)   |
|  OS   | Block Device | "physical" sector size (4KiB: 4kn or 512e physical block device) |
|-----------------------------------------------------------------------------------------|

1) Legacy Guest OSes (MS-DOS and Win9x) have a standard offset of 63 sectors (63*512B sectors = 31.5KiB) for the first partition. This offset is not an integer multiple of 4KiB and causes misalignment of the Guest's logical allocation unit with a modern 4KiB aligned Host block device. This Guest-to-Host misalignment causes write-amplification on the Host due to the Guest logical allocation units (FAT clusters) partially overlapping two "physical" sectors on the Host block device.

2) Even when Guest and Host files systems and block devices are aligned, write-amplification can still occur when the Guest FS uses an allocation unit that is smaller than the Host FS's allocation unit. For example: A 32MiB Guest block device with FAT16 and default 512B cluster size and a Host NTFS FS with default 4KiB cluster would cause the Host to exhibit 8X write-amplification (8x read-modify-write cycles for the same physical sector) for sequential writes to the Guest virtual block device. Constraining the Guest FS's cluster size to the same value as the Host's logical allocation unit reduces write amplification to a minimum. That said, any integer multiple of the Host's allocation unit is valid for the Guest cluster size but reduces Guest-side storage efficiency for small files and can lead to additional write-amplification on the Host FS: A partially filled Guest cluster still writes to all Host logical allocation units it is composed of. KISS: use the same size for both.

POTENTIAL SOLUTION: Feng Shui

I believe the following constraints would ensure proper alignment of a legacy Guest FS and modern 4KiB aligned Host block device.

For small virtual block devices use 32KiB alignment:

1)  Start first FAT partition at sector 64 (on an emulated 512n block device)
2)  Constrain FAT partition sizes to a multiple of this offset (partition_size = n*64*512B)
3)  Constrain FAT partition cluster size to 4KiB or more

This minor change keeps things aligned Host side and has minimal loss of Guest storage space.

For large virtual block devices use standard 1MiB alignment:

1)  Start first FAT partition at sector 2048 (on an emulated 512n block device)
2)  Constrain FAT partition sizes to a multiple of this offset (partition_size = n*2048*512B)
3)  Constrain FAT partition cluster size to 4KiB or more

This matches modern defaults.

In either case, to minimize Host-side write-amplification, the Guest FS cluster size should match the Host FS cluster size.

When testing Win98SE on DOSBox-X (CrystalDiskMark and ATTO benchmark), I didn't notice much if any performance difference between standard 63 and 2048 sector offsets with FAT32 and 4KiB cluster size on a 1024MiB emulated block device. But it doesn't hurt to use best practices when making virtual block devices that might be shared and used with QEMU, which has a much faster block device subsystem.

QUESTION: TLDR; Can IMGMAKE Learn New Tricks?

Would it be possible to add these profiles as an option to IMGMAKE in DOSBox-X? Say "-h 4kib" where parameter "-h" refers to "host" and argument "4kib" implies 4KiB alignment of Guest block device. I don't rightly know if these settings are fully compatible with all DOS flavors and FAT versions (e.g., is 4KiB cluster size always a valid option?). But for Win9x these defaults should work fine. Thoughts?

Have you checked that no similar question(s) exist?

Code of Conduct & Contributing Guidelines

the-burrito-triangle commented 15 hours ago

For reference, here's a good write up on this subject by Thomas Krenn: https://www.thomas-krenn.com/en/wiki/Partition_Alignment_detailed_explanation