Measuring Disk I/O

[ross-spencer]

I don't have such a great instinct for these things, there are Latency Numbers Every Programmer Should Know and I recognize disk seek and disk i/o is slow which helps me at least design my apps to minimize this.

I don't think xattrs exist on the chart above.

• Do we know if xattrs use disk i/o?
• How do they compare to disk i/o?
• How can it be measured?

Folks might have answers?

When we get to writing docs about this, I feel like a reference to this in those could be helpful/useful/interesting for others to read about.

[pjotrek-b]

Good question(s)! :smile: Hm...

A quite detailed article on performance considerations and code: "A Distributed Key-Value Store using Ceph" (by Eleanor Cawthon, 2012)

I haven't read it yet myself, but from a quick peek it looks interesting? It even explicitly mentions xattrs - and Ceph seems to be designed to use cache/access databases for better performance. https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/

And Ceph has OMAP as key-value database:

• https://medium.com/opsops/rados-objects-omaps-and-xattrs-32e66d2b528b
• https://ceph.io/en/news/blog/2015/get-omap-keyvalue-size/

To reply to your questions:

1. I assume a clever fs implementation will cache at least recently read xattrs somewhere?
2. Since limit-values of different filesystems (ext4, btrfs, BeFS, etc) seem multiple of 4 kiB, I assume it's the same "reading a few blocks" i/o as for any other file data. I honestly don't know.
3. I'd time code-execution for different actions (get/set, single key, multiple keys, etc).

Would actually be interesting if there actually /is/ an existing "xattr-benchmark" code?

[pjotrek-b]

Maybe interesting quote from Apache Hadoop's official docs on Extended Attributes (xattrs) on HDFS

"HDFS supports extended attributes out of the box, without additional configuration. Administrators could potentially be interested in the options limiting the number of xattrs per inode and the size of xattrs, since xattrs increase the on-disk and in-memory space consumption of an inode."

dfs.namenode.xattrs.enabled Whether support for extended attributes is enabled on the NameNode. By default, extended attributes are enabled.

dfs.namenode.fs-limits.max-xattrs-per-inode The maximum number of extended attributes per inode. By default, this limit is 32.

dfs.namenode.fs-limits.max-xattr-size The maximum combined size of the name and value of an extended attribute in bytes. By default, this limit is 16384 bytes.

[pjotrek-b]

Re-discovered this article (which is quite recent: 2024-05-13) https://eclecticlight.co/2024/05/13/apfs-extended-attributes-revisited/

Quote:

For smaller extended attributes up to 3,804 bytes, their data is stored with the xattr in the file system metadata. Larger extended attributes are stored as data streams, with separate records, but still separately from the file data. Apple doesn’t give a limit on the maximum size of xattrs, but they can certainly exceed 200 KB, and each file and folder can have an effectively unlimited number of xattrs.

This reads to me as if care has been taken to store the metadata (smaller) somewhere else (for faster, easier access?) - and apfs already takes care of this in the background. I'd really like to do benchmarking on xattrs already.

And it also sounds pretty awesome, for future creative use of xattrs (unlimited? wohooo! :sunglasses: )

Do you have time to setup/code at the moment? It'd be more fun, and easier not having to do that on my own :)

[pjotrek-b]

...reading that article, I feel like APFS might become my annotation-storage goto filesystem of choice. Yet, I believe that btrfs may serve me just as well - if not a even i bit better.

Anyways: We've got plenty of existing "carpet" to build AHAlodeck on. As xattrs seem finally to be as widely supported already to actually "use them for what they were invented for" :grin:

I feel like a kid in a candy store! Yesterday I discovered that F2FS allows going Android with xattrs - full roundtrip already (not tested yet). Wow!

Considering flash cards on mobile devices: I/O may actually be a really good question. I do however believe, that "the consortium" that commissioned development of F2FS knows what they're doing - and that's all high-throughput on nand stuff.