search

FZJ-JSC / JUBE

The JUBE benchmarking environment provides a script based framework to easily create benchmark sets, run those sets on different computer systems and evaluate the results. It is actively developed by the Jülich Supercomputing Centre of Forschungszentrum Jülich, Germany.
https://www.fz-juelich.de/ias/jsc/jube
GNU General Public License v3.0
34 stars 12 forks source link

jube continue has quadratic complexity without specific purpose #4

Open jprotze opened 1 year ago

jprotze commented 1 year ago

jube continue seems to check done/error files in all work package directories while processing one work package (according to strace). This results in at least 2n^2 file accesses when only 2n should be necessary. For a larger experiment space and slow meta data server, this can result in significant execution time.

wellmann99 commented 1 year ago

I had a quick look at the problem using a few examples. It turns out that the done and error file check happens more often than expected and needed, and even more often than described here. However, due to the complexities involved, we need a little more time to analyse and fix the problem.