search

hpc / ior

IOR and mdtest
Other
377 stars 166 forks source link

MDtest stonewalling ignored for stat phase #134

Closed JulianKunkel closed 4 years ago

JulianKunkel commented 5 years ago

When invoking mdtest with stonewalling and running in phases, the stat phase does not read the number of files created in the create phase.

Code to reproduce the bug: Create the files: $ mdtest -C -W 1 -F -d out -n 2400000 -u -L -x stonewall-file.txt Stat the files: $ mdtest -T -F -d out -n 2400000 -u -L -x stonewall-file.txt

Changes to stonewall-file.txt appear to be ignored.

Bug was reported by Sven.

JulianKunkel commented 5 years ago

Another way of testing: $ mdtest -C -W 1 -F -d out -n 10 -u -L -x stonewall-file.txt $ mdtest -T -F -d out -n 100 -u -L -x stonewall-file.txt

An intermediate fix is here: https://github.com/hpc/ior/pull/new/fix-134 Side effects need to be checked carefully as the mdtest code is complex with all possible branches. Likely that some case beyond IO-500 is not covered properly.

Best is to rewrite the code from the recursive and complex model and extract the file name creation from the actual execution engine.

cryptosven76 commented 5 years ago

hi, any update on this one ?

JulianKunkel commented 5 years ago

Not yet, I focused on other issues. As said, the best fix for this is a rewrite for the way data is processed as any (quick) fix will likely cause new problems.

Am Do., 11. Apr. 2019 um 17:05 Uhr schrieb cryptosven76 < notifications@github.com>:

hi, any update on this one ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/hpc/ior/issues/134#issuecomment-482177448, or mute the thread https://github.com/notifications/unsubscribe-auth/AE1uyq9Of8cLounYsHqe0DiXAck-u7oWks5vf107gaJpZM4bDv1R .

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/ PGP Fingerprint: 1468 1A86 A908 D77E B40F 45D6 2B15 73A5 9D39 A28E

johnbent commented 5 years ago

Will this be a problem for IO500 or are we pulling from the branch where you have this fixed? We will need to merge in the other large pull request I just made to IOR and mdtest to fix shifting.

JulianKunkel commented 5 years ago

We can merge your changes, ignoring such branches.

Am Sa., 27. Juli 2019 um 22:56 Uhr schrieb John Bent notifications@github.com:

Will this be a problem for IO500 or are we pulling from the branch where you have this fixed? We will need to merge in the other large pull request I just made to IOR and mdtest to fix shifting.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

-- Dr. Julian Kunkel Lecturer, Department of Computer Science +44 (0) 118 378 8218 http://www.cs.reading.ac.uk/ https://hps.vi4io.org/ PGP Fingerprint: 1468 1A86 A908 D77E B40F 45D6 2B15 73A5 9D39 A28E

karthjyojay commented 2 months ago

Is this in the main branch? I have noticed this behavior still.