Questions Regarding Software Detection

[younghyunc]

Hi all,

I am working on an autotuning project (called GPTune) for HPC applications. We are also interested in providing an automatation interface based on CK, which we call CK-GPTune, as an additional interface for GPTune.

I have two questions about the CK soft detection module:

• What would be a good way to detect software which is located in a very deep directory hierarchy? Increasing the limit_recursion_dir_search value adds more search time. Adding extra_search_dirs requires more human efforts (and different machines might have different paths). Can the ck-soft module have more default search directories depending on the type of software (e.g. including sys.path output as default search for Python modules?)?

• "soft_file" in software meta description: It seems both Linux and Mac use the "soft_file" name in the "linux" key in the meta description dictionary. If the software can have different names (or different file extension names) on Linux and Mac, what would be a good way to detect them?

Thank you,

Younghyun

[gfursin]

Hi Younghyun,

Thank you for your interest!

Regarding your questions: 1) There is a way to customize search directories for a given software using "soft" customization plugins. Here are a few examples of such customization:

• OpenCL lib
• Android NDK
• Python SciPy lib

If I remember correctly, the "dirs" list contains the default search directories and you can substitute it completely.

If this logic is not enough, we can always update the CK "soft" module. Here is the place where we assemble a list of search paths depending on the operating system, host/target platform and other parameters.

2) You are right, "soft_file" does not differentiate between linux and macos (they considered as Linux dialect). If it's needed, we usually search for some file from a given soft that has the same name in both Linux and MacOS. Then we specialize the environment and other file names depending on a host/target OS (Linux vs MacOS) via the customize.py plugin.

A few examples:

• GCC detection: source 1, source 2, source 3
• ICC detection: source 1, source 2

Hope it's of any help! GPTune project sounds cool - good luck and thanks again for your interest!

CC @phesse001 @joy369

[younghyunc]

Thank you for your reply. These are very helpful. I have resolved my issues by patching the CK's soft module and having separated meta.json files for Linux/Mac.

Also, thank you very much for providing the CK framework! We are interested re-using (expensive) performance data obtained from multiple HPC centers. The CK's technology and ideas will be very helpful to our framework too.

[gfursin]

Glad to hear that, @yhcho614, and thank you for your nice words - I am really glad if CK was useful!

I expect to get some funding to continue improving the CK technology soon and will keep in touch about that!