The filesystem did't record the space of special vdev,but record the metadata size.

[yuanfangcc]

Describe the feature would like to see added to OpenZFS

zpool list -v pool_10kw

NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT pool_10kw 1.48T 68.0G 1.41T - - 0% 4% 1.00x ONLINE - sdb 1016G 47.7G 968G - - 0% 4.69% - ONLINE special - - - - - - - - - sdc 496G 20.3G 476G - - 2% 4.09% - ONLINE

zfs list pool_10kw/fs

NAME USED AVAIL REFER MOUNTPOINT pool_10kw/fs 67.9G 885G 67.9G /pool_10kw/fs

If a special vdev is added, metadata is stored on the special vdev. The size of the file system is not recorded in the special vdev, but the usage of the file system is included in the special vdev.

( 67.9 + 885 ) < 1016 ;The total size is the data vdev space. 67.9 = 47.7 + 20.3 ; The usage is the total usage of the data vdev and special vdev.

Is that unreasonable?

I think the file system usage should only record the data vdev usage space. Otherwise, add the metadata vdev does not increase the file system space.

[rincebrain]

It's not included in the free space otherwise you could wind up with the situation where you get told you have, say, 100G free on the special vdev, and df says 100G free, but you get ENOSPC trying to write.

You neglected to specify the ZFS version you're using, but that amount free in zfs list seems about right for 3.2% reserved space and 48 GiB allocated on the data vdev.

It's possible it's including the special's size in the 3.2% calculation, depending on the precise numbers involved, which it probably shouldn't, though the ceiling on slop space on newer versions will mean in practice that rarely matters, I think...

[yuanfangcc]

I know that the file system should not record the special vdev size, which may cause the usage space to exceed. I wonder if the file system can only record the usage of data vdev and not the usage of metadata disks. Even if the special vdev is full and metadata is stored on the data vdev, the file system normally records the usage of the data vdev. Is this feasible?

[rincebrain]

I don't think that would be a good idea.

For much the same reason as I think raidz reporting less than the logical used size for files that are incompressible in some cases (see #14420), I think hiding some of the space usage in USED when it goes onto the special vdev would lead to people being much more surprised.

Imagine if you wrote 1 GiB of 4k records that all ended up on the special vdev, so it doesn't show up in USED or AVAIL. Most people would assume the data didn't actually get written.