Non-deterministic ordering of pmc_perf_*.txt file (sometimes???)

ROCm / rocprofiler-compute

Advanced Profiling and Analytics for AMD Hardware

https://rocm.docs.amd.com/projects/omniperf/en/latest/

MIT License

135 stars 49 forks source link

Non-deterministic ordering of pmc_perf_*.txt file (sometimes???) #332

Closed skyreflectedinmirrors closed 3 weeks ago

skyreflectedinmirrors commented 7 months ago

https://github.com/ROCm/omniperf/blob/59254a50463a8f3fe44c6b1f06610edf53e2c30c/src/omniperf_profile/profiler_base.py#L90

RE: our internal debug, we noticed counters from a pmc: line in a file, occasionally get split over multiple runs in pmcperf<#>.txt. This results in nonsensical values sometimes, due to-to-run variation.

Need to debug how this could happen.

Seems like some sort of non-deterministic ordering in a python API / pandas / etc.

ppanchad-amd commented 3 weeks ago

@skyreflectedinmirrors Issue is fixed. Closing ticket. Thanks!