khuck
commented
1 year ago The APEX Idle timer is not a rate or percentage, it is an accumulated time. I recently reformatted the output and did not align those columns in the right place to line up with the other timers, sorry for the confusion.
A few other observations:
Total processes detected: 32
HW Threads detected on rank 0: 128
Worker Threads observed on rank 0: 128
Available CPU time on rank 0: 2166.28 seconds
Available CPU time on all ranks: 69321 secondsWhat this part of the report is telling you is that you are using 128 threads per process, and 32 total processes. That's fine, but you're likely using hyper threading which may or may not be more efficient than running without hyper threading.
The timer report is aggregated and averaged across all ranks. The total main time represents all time from when rank 0 started until it ended, and APEX assumes all processes had the same lifetime.
If you want the data from all ranks and you are using apex_exec, you can add the --apex:csv flag. If you are not using apex_exec, set the environment variable export APEX_CSV_OUTPUT=1 before running. Then you will get a file called apex_profiles.csv which contains all the performance data from all ranks. You can post-process that data with the apex-summary.py python script.
JiakunYan
I am running Octotiger + HPX + APEX with 32 ranks, and I found the idle rate is unreasonably high. Here is the output:
The idle rate is
99.1%.After looking at the code of APEX, I found the following lines:
I don't understand why APEX chooses to multiply the
total_mainwith rank num. I didn't find any evidence that APEX aggregates timers across ranks, does it?Another quick question: is there a way to get the report from all HPX ranks, instead of just rank 0?