Caliper overwriting output file?

[cielling]

I have a (Fortran) code that has been instrumented with Caliper and is configured so that it can trigger Caliper to write a report at regular intervals through the 'flush' method on the config-manager.

If I configure Caliper at runtime to collect the runtime-report and report it at, say, every 5 cycles to stdout (i.e. the default output destination), I get a runtime-report written to stdout every 5 cycles as expected.

However, if I set Caliper to write the runtime-reports to a file instead of stdout (with the 'output=my_output.txt' option), the final file that is written only contains the last runtime-report, and not any of the previous ones that should have been emitted during the run. It looks to me that Caliper is overwriting the file every time it writes a report.

I would have expected the behavior to be the same, regardless of the destination of the output. Is there an 'append' option that I'm missing? Or is this a bug in Caliper?

Thanks for any help!

[daboehme]

Hi @cielling, there is currently no append option unfortunately.

You can however set a Caliper attribute and use that to create a new filename for every flush. That may be a workaround. The example below shows how to set an attribute "cycle" for the flush. You can then set output=my_output_%cycle%.txt for the filename, and it should create my_output_5.txt, my_output_10.txt, etc.:

call cali_begin_int_byname('cycle', i)
call mgr%flush
call cali_end_byname('cycle')

Hope this helps!

[cielling]

Hi @daboehme, thanks for the response!

So what I'm hearing is that output to a file is at least somewhat intentionally different from output to stdout?

Would Caliper be able to provide us with a flag to append the outputs when writing to a file?

The concern that the code team has with creating a new file for every flush is that this could potentially create thousands of files which would be stressing the file system.

[daboehme]

Hi @cielling, yes the output to file is different from the stdout/stderr output.

I think it would make sense to offer an append option though and even make that the default. I'll look into it.

[cielling]

@daboehme Having an append option would be great. Thanks for looking into this!

[cielling]

Hi @daboehme, I wanted to check in to see when you/the Caliper team will have time to look into the append option? Thanks!

[daboehme]

Hi @cielling, we now have the option to append to a file. It is default on for runtime-report so you don't have to do anything to enable it.

[cielling]

Thank you, @daboehme, much appreciated!