Support files-per-dir by disabling stonewalling.

JulianKunkel commented 3 years ago

Note this is only allowed if needed and the committee approves.

wszoecs commented 3 years ago

Just a feedback on this fix: YES it works as envisioned. (although - its practical usage is a bit tricky)

Here just some more detail - why this is required:

The E1000-Lustre-MDSs do support DNE-phase2 (striped dirs) The problem is only, that due to the file-naming - ldiskfs doesn't allow me, to create more than ~8mill files/dir. (see for a more detailed explanation: https://github.com/IO500/io500/issues/15 --- issue-#15) S O - with let's say 2 dirstripes - we can then have ~16 mill files in that directory. But - just with the right node-count - I get around 120k file/creates/sec The math then for '-n' and '-I' is:

let's say with 192 nodes 20 ppn (3840 procs) and target 300+ sec runtime.
300sec * 120k = ~36000000 (so 36mill files is the estimate)
16mill (dir limit) multiples are 16m,32m,48m ...
So there's no real 'close' 2-multiple for 36m.
so let's try with 3: 36mill / 3 = 12mill files per dir_loop
The math is then: 12mill / 3840 procs = 3125 - this is your '-I' argument (files_per_dir)
Now '-n' (if '-I ' is present) only can be a multiple of '-I' - and we estimated '3' loops: "-n 9375"
actually - I've chosen "-n 9900 -I=3300" - and got 310.374 sec runtime.

Again - the '-I=X' is required to not run out of file-entries in a shared dir. Using this option - does not work with stonewalling (it only does 1 dir_loop if stonewalled). Turning off stonewalling - requires this manual calculation - to not over-create too many files. (too many files like 48m instead of 36m - will penalize your find score quite a lot)

In general - the proposed fix worked for me fine.

JulianKunkel commented 3 years ago

Thanks for the extensive description of the situation.

adilger commented 3 years ago

You could enable the large_dir feature for ldiskfs, which fixes the directory entry limit.

gflofst commented 3 years ago

changing to enable large_dir was what was required previously. What is the motivation for changing this policy.

JulianKunkel commented 3 years ago

After a discussion with the board: I will extend the patch to mark the submission automatically as invalid and show this in the options.

JulianKunkel commented 3 years ago

The respective change is made.

IO500 / io500

Support files-per-dir by disabling stonewalling. #41