search

RRZE-HPC / kerncraft

Loop Kernel Analysis and Performance Modeling Toolkit
GNU Affero General Public License v3.0
88 stars 24 forks source link

Improve handling of intermediate files #56

Closed cod3monk closed 5 years ago

cod3monk commented 6 years ago

The goal is (and always was) to allow the user intuitive and transparent access to intermediately generated files (namely a compilable C version for IACA analysis, an assembly version for IACA analysis, a compilable C version with LIKWID marker API for execution). The current approach is to create a $kernel_file_name.c_compilable.c for any compilable C code (either for IACA analysis or execution with LIKWID) and a $kernel_file_name.c_compilable.s for assembly files. The c file will be different, depending on the performance model last run.

I'm in search for a cleaner way to do this, especially one that is non-ambigues about the origin and purpose of the intermediates and one that does not clutter the kernels directory.

rrzeschorscherl commented 6 years ago

One option would be to generate, with each kerncraft run, a new folder with a unique name in $PWD (or in another place if a suitable option is given). All generated files could go there, and they would be safe from being overwritten in subsequent runs. A complete log of the output could also be stored in this folder. Folder names could be something like ./kanalysis`date +%s`/ or include a random number to be super safe. Best of all, the data can be deleted with a simple command.

Other performance tools (especially tracing tools) have the same problem, and they use similar solutions.

cod3monk commented 6 years ago

How about the following behaviour: By default, do not preserve any intermediate files (use a safe temp directory and delete after completion). If --intermediate or -i is given, create a directory (e.g., original_file_name_wo_ext.c.KC#/ with a serial number #) and keep a copy of intermediate file, including a log of inputs and outputs (with highest verbosity).

Currently we have the following intermediates: (1) plain compilable C code --- original_file_name_wo_ext.compilable.c (2) plain assembly file generated from (1) --- original_file_name_wo_ext.compilable.s (3) assembly file with IACA markers, based on (2) --- original_file_name_wo_ext.compilable.iaca_marked.s (4) binary object file with IACA markers, generated from (3) --- original_file_name_wo_ext.compilable.iaca_marked.o (5) compilable C code with LIKWID markers --- original_file_name_wo_ext.compilable.likwid.c (6) binary executable file with LIKWID markers, generated from (5) --- original_file_name_wo_ext.compilable.likwid

cod3monk commented 5 years ago

Kerncraft now stores all intermediate files in .<kernelname.c>_kerncraft. With the optional argument --clean-intermediates they are deleted after use. These files also act as a cache, which may speed up scaling runs significantly.