CK detection of some software is extremely slow on NFS (noticed at LLNL)

ctuning / ck-env

CK repository with components and automation actions to enable portable workflows across diverse platforms including Linux, Windows, MacOS and Android. It includes software detection plugins and meta packages (code, data sets, models, scripts, etc) with the possibility of multiple versions to co-exist in a user or system environment.

https://github.com/mlcommons/ck

BSD 3-Clause "New" or "Revised" License

72 stars 25 forks source link

CK detection of some software is extremely slow on NFS (noticed at LLNL) #82

Closed gfursin closed 5 years ago

gfursin commented 6 years ago

When preparing a demo workflow for SCC18 (https://github.com/ctuning/ck-scc18) with #CollectiveKnowledge, @spack and @flux-framework during my visit to @LLNL, we've noticed that CK software plugins (http://cknowledge.org/shared-soft-detection-plugins.html) work extremely slowly on NFS, i.e. 3..4 minutes to detect one software dependency.

This is normal since we perform the full and recursive search of all directories by default, however maybe we should provide a flag like --quick to speed up search in such case by looking only in a few most commonly used locations. We can then let a user specify the full path to a given soft if it's not found?

CC @SteVwonder @trws @tgamblin

trws commented 6 years ago

Some tricks we to use for working around this in our environment (in case you may find them useful):

as @tgamblin mentioned in the meeting: do a search across all paths in parallel, this tends to help with overall throughput
leverage pkg-config where possible, its search path tends to be shorter and can give a more complete picture than just where the binary or library is
hook into the path cache of a shell, for example the $commands special associative array in zsh stores the presence and location of every command it found on startup, so a check for existence with zsh -c 'exit $+commands[]' can be surprisingly fast, especially if you want to look for multiples or through a long path.

gfursin commented 6 years ago

Thanks @trws ! Yes, we can try parallelization at some point - we have one search function used across most CK modules so it should be doable. As for using full search - we originally did it because we had to support both Linux and Windows, and didn't have time to specialize it. However, since it's stable now, we can probably provide such a specialization for Linux-based machines.

gfursin commented 5 years ago

Hi all! While at SC18, I fixed that with @SteVwonder . I added a CK kernel var to substitute default search directories for soft (/usr, $HOME, etc):

$ ck pull repo:ck-env
$ ck set kernel var.soft_search_dirs=dir1,dir2,...

Just substitute above dirs with something reasonable for your environment. It worked fine and we can even point it to the directory with spack packages!