Open sfantao opened 9 months ago
There are a couple things going on here. I believe the default view of the timelines is the accumulation of the counters, so you will not see them fluctuate but instead, grow over time — if you click on the lightning bolt looking thing, you can change the view, I think one of them will be the delta. Second, there are likely some discrepancies from mapping hardware counters for kernels onto the kernel-independent timeline. Third, I don’t have a ton of confidence in the combination of the timing alignment between omnitrace’s current use of roctracer for kernel timing with the kernel timings reported by rocprofiler when it reports the HW counters — this needs to be investigated.
Using as an example
https://github.com/amd/HPCTrainingExamples/tree/main/HIPIFY/mini-nbody/hip
, if I get device counters with rocprof using:I get:
If I use omniperf with a configuration containing:
and run:
I get:
i.e the counters do not show any fluctuation as they should trusting the rocprof output.
Tested on
ROCm 5.7.0
and omnitraceomnitrace-1.10.4-ubuntu-20.04-ROCm-50700-PAPI-OMPT-Python3.sh
.For completeness on different machine and ROCm 5.6.1 I see things like:
Also no fluctuations but for the first kernel the reading starts correct but shifts in the middle of the kernel.