search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.61k stars 1.75k forks source link

Ashift=12 Pool: Significant Storage Wastage for Small Files #15846

Open sidalo opened 9 months ago

sidalo commented 9 months ago

Unzipping the following 44MB file will require about 243MB of storage on a single-disk Ashift=9 pool (340MB if using Raidz). http://mirrors.rit.edu/gentoo/snapshots/portage-latest.tar.xz

If it is on a single-disk Ashift=12 pool, it will take about 750MB of storage space. If the Ashift=12 pool is of Raidz type, it will require about 1.5GB of space.

Considering storage space efficiency, Ashift=9 should be considered, even for 4k sector drives.

However, in the current version of zfsutil, creating an ashift=9 Raidz type pool for 4k sector drives is not allowed. I need to uninstall Zfsutil and install Zfs-fuse to force the creation of an ashift=9 pool for 4k sector drives. An option for creating ashift=9 should be allowed.

IvanVolosyuk commented 9 months ago

I made a size distribution of my copy of gentoo portage file sizes, where the first number is size of file in bytes, second number is count of files less or equal than that size, excluding those in a previous bucket. 112: 29610 512: 40369 1k: 36067 2k: 23378 4k: 12926 8k: 3549 16k: 1386 larger: 719

Files less than 112 bytes after compression can fit into block pointer, but larger than that will use a-shift-sized block. Portage constitutes less than 1 percent of my data, so I don't feel pressed for space with this inefficiency.

I made the following experiment on my 12 ashift ZFS pool with lz4 compression:

truncate -s 10G /foobar
zpool create -o ashift=9 portage /foobar
cp -av /usr/portage /portage
zpool export portage

The resulting /foobar file size is only 112M, while the sum of all file sizes in my portage tree is equal to 179M. So, if you are constraint on disk space the nested ZFS can be a workaround for you.

GregorKopka commented 9 months ago

Is pool-on-pool still a recipe for disaster or have the (IIRC) deadlocks in that situation been fixed?

IvanVolosyuk commented 9 months ago

It might cause deadlock on export if parent pool is exported first I guess. I had issues with zvol backed nested pool - something with wrong blocksize and deadlock. But for file backed pools I haven't seen any issues.

sidalo commented 9 months ago

The other way to create ashift=9 pool is by Hardware Raid (maybe can be done with mdadm). Each of them to be Raid 0 then create an ashift=9 pool. Since the Raid card operates at 512B sector the zfsutil would not force it to work at 4k sector. From my benchmark results between 512B (ashift=9) and 4k (ashift=12) are the same.

perccli /c0 add vd each r0 drives=32:0,1,2,3,4,5 (0 of "c0" is my raid controller number, 32 is my enclose number, 012345 are the slot number on the enclose.) zpool create -o ashift=9 POOLNAME raidz2 /dev/sd[a-f]