search

flux-framework / dyad

DYAD: DYnamic and Asynchronous Data Streamliner
GNU Lesser General Public License v3.0
7 stars 5 forks source link

Fix performance bottleneck in cmp_canonical_path_prefix #87

Open ilumsden opened 8 months ago

ilumsden commented 8 months ago

While evaluating DYAD with the workflow benchmark, we found that cmp_canonical_path_prefix had a surprisingly large cost in terms of time. Some fixes, as suggested by @hariharan-devarajan, are:

  1. Remove the use of realpath
  2. Switch to a prefix tree-based algorithm for path comparison
JaeseungYeom commented 8 months ago

You can cache the result of the first realpath() call as it translates the dyad managed path to real path. The second one is there for correctness. You cannot omit that.

hariharan-devarajan commented 8 months ago

We also have to switch strcmp to prefix tree for faster comparison.

JaeseungYeom commented 8 months ago

PR #89