amotin
commented
1 month ago While it should not be difficult to implement it for different vdevs, we would mostly have to specify via vdev property which L2ARC is which and accept more hardware and administrative overhead, which I am not happy about. Implementing it within one device might be slightly more invasive, but should be easier to manage, though setting limits manually might be tricky. And I generally don't believe in required manual tuning in production scale. I think we should first focus on existing issues of the L2ARC implementation I listed several times and then think how to better balance data and metadata writes after that.
tkittich
2TAC
Describe the feature would like to see added to OpenZFS
To cache more metadata in L2ARC, perhaps L2ARC should be split into non-overlapping metadata L2ARC and data L2ARC. Data buffers would go to data L2ARC, and metadata buffers would go to metadata L2ARC.
How will this feature improve OpenZFS?
The goal is to increase L2ARC hit rate by storing most metadata in L2ARC. With this feature,
l2arc_mfuonly = 2, and persistent L2ARC, most metadata should eventually be in L2ARC. That should increase the L2ARC hit rate to be way more than 50% (assuming 1 metadata hit per 1 data hit). The size of metadata L2ARC should be automatically and dynamically set to be large enough to store most metadata of the pool. Data L2ARC can take the rest of the cache device. This would essentially make a metadata L2ARC somewhat similar to a special vdev.Additional context
With
l2arc_mfuonly = 2and #16499 or #16648, there's still no guarantee that most metadata would get stored in L2ARC. Data buffers could get written into L2ARC so fast that they would overwrite metadata. Setting aside enough space for metadata in L2ARC would make the guarantee.