nerscadmin / IPM

Integrated Performance Monitoring for High Performance Computing
http://ipm-hpc.org
GNU Lesser General Public License v2.1
81 stars 35 forks source link

Is it possible to trigger profiling output before program exit? #30

Closed skyreflectedinmirrors closed 6 years ago

skyreflectedinmirrors commented 6 years ago

I'm trying to profile different sections of a solver in OpenFOAM, on small cases this works quite well.
However for large cases I need tens of runs on the cluster to complete my case.

I would (ideally) like to put a call like:

ipm_output();

at the end of my time-loop (every time-step can take hours to complete) such that I have a record of the time spent in my defined sections.

Alternatively, I would have to implement some sort of interrupt signal for OpenFOAM to exit the program once I near the end of my time-reservation which seems to be difficult

skyreflectedinmirrors commented 6 years ago

Looking into this a bit more, it seems like I can ensure output at the end of reservation by changing my SLURM batch file to send a SIGTERM on exit via --signal (which IPM should catch). I'll close this issue