problame
commented
1 year ago Is your solution already implemented somewhere?
No, it's not implemented anywhere yet.
Would it change something in this case? I assume so...?!
Your description is unclear about how the normal allocation class looks like, i.e., what are you doing with the HDDs. A zpool status dump says more than a thousand words :)
Also, provide the full fio config / command line for each result. For example, it's unclear whether you are doing the randwrites with --sync or not.
Finally: I don't have experience with special vdevs.
The docs say:
By default this includes all metadata, the indirect blocks of user data, and any deduplication tables. The class can also be provisioned to accept small file blocks.
So, with your 256k writes, the special devices are only used for ZFS metadata. Which means,
- without a ZIL your bottleneck is most likely the HDDs, regardless of whether they are striped / raidz'd / draid'd
- with a ZIL
- if you use
logbias=throughput, ZFS will use WR_INDIRECT log recordsi.e., it will issue the data to the final location (on the HDDs) and only store a pointer to the data in the ZIL. Which again, makes you bottleneck on the HDDs. - Learn more about it in this comment https://github.dev/openzfs/zfs/blob/bf1bec394e715de9f7ac4a54b5b1b884d7e41a32/include/sys/zil.h#L394-L423
- without
logbias=throughput - for the NVMe-mirrored ZIL pool, the
1237MiB/slooks like typical ZFS overhead over the1849MiB/sbaseline - the PMEM-mirror ZIL throughput is emberassing, but, I'm not completely surprised. I don't have access to PMEM hardware anymore, though, so, I can't confirm the measurements.
- if you use
thomasklosinsky
Hi Christian,
I have seen your presentation of the zil-pmem method (https://www.youtube.com/watch?v=2ouRsQEbusk) and I am looking for ideas why a new storage server (Supermicro SSG-640SP-E1CR60) is behaving like this:
fio on Intel Optane 200 128GB Direct bs=256k [w=1250MiB/s][w=4998 IOPS] fio on Micron 7400 1.92T NVMe Direct bs=256k [w=1849MiB/s][w=7395 IOPS]
which already is weird.
Latency is:
PMEM 8us NVMe 80us
HDD Pool contains 4x 12x Seagate Exos 16T
With 256k recordsize (and 256k blocksize) tested with fio, I get the following randwrite results:
sync=standard - no ZIL - 2x NVME mirro special VDEV = [w=9225KiB/s][w=36 IOPS] sync=standard - PMEM mirrored ZIL - 2x NVME mirro special VDEV = [w=547MiB/s][w=2189 IOPS] sync=standard - NVMe Mirrored ZIL - 2x NVME mirro special VDEV = [w=1237MiB/s][w=4949 IOPS] sync=disabled - ZIL not used - 2x NVME mirro special VDEV = [w=3721MiB/s][w=14.9k IOPS]
Of course, we would love to use sync for a storage server ;)... but could not believe that the PMEM result is that bad...
Is your solution already implemented somewhere? Would it change something in this case? I assume so...?!
Best regards, Thomas