MDTest hard on Lustre

[JulianKunkel]

This issue is to document and discuss the known problem of running MDtest hard on Lustre which may abort due to a "No space left on device" error. The reason is that by default, with LDiskFS: "We use the ext3 hashed directory code, which has a theoretical limit of about 15 million files per directory". With a deployed DNE2 or a specific mount option, this limit can be avoided.

[adilger]

There are two mechanisms to increase the ~15M files-per-directory limit for Lustre, both of which are orthogonal and can be used at the same time:

• in the io500.sh script's setup_directories function, use "lfs setdirstripe -D -c -1 $io500_workdir" to set the default directory striping for all subdirectories created therein. This will cause all subdirectories created under $io500_workdir to be striped over all available MDTs, which is appropriate for very large directories such as mdtest-easy and mdtest-hard and does not hurt the ior-easy or ior-hard results. This option is useful only if the filesystem has more than one MDT configured, requiring Lustre version 2.8 or later on both the client and server, though preferably Lustre 2.12.5 or later. This is also recommended for ZFS MDTs.
• enable the large_dir feature for all ldiskfs MDTs (this must be done by the administrators via "tune2fs -O large_dir /dev/<mdtdev>" for each MDT). This feature requires at least e2fsprogs-1.44.3.wc1 installed on the MDT(s) to enable this feature, though it is always recommended to use the latest version from https://downloads.whamcloud.com/public/e2fsprogs/latest/ (e2fsprogs-1.45.6.wc2 at the time of writing). This feature is enabled by default for ldiskfs MDTs formatted with Lustre 2.14 and later, but is not needed for ZFS MDTs.

[adilger]

Of course, it is also possible that the MDT(s) may actually run out of space, if the creation rate is high, and there is not enough free inodes on the MDTs, or if Data-on-MDT is not used for the small files and there are not enough inodes on the OSTs. The free and used inodes can be checked by running "lfs df -i" on any client.

[wszoecs]

The issue with 'mdt-hard' (shared dir file-create) - is abs. not in 'large_dir' - but the way how ldiskfs does directories: It uses HASHED buckets - to place the files. And due to the hilarious naming of mdtest - your files all end in 1-2 maybe 4 buckets - and fills thos up. So - even if you still have several millions of entries available - you're running out of available entries in a few hash-buckets. my experiments say - per MDS - at around 8 mill - you're DONE - and always/reliably run out of space in this test.