ROCm / rocprofiler-compute

Advanced Profiling and Analytics for AMD Hardware
https://rocm.docs.amd.com/projects/omniperf/en/latest/
MIT License
135 stars 49 forks source link

Running `omniperf profile` after `omniperf profile --roof-only` erases all roofline pdfs #202

Open gsitaram opened 12 months ago

gsitaram commented 12 months ago

Describe the bug If we profiled the application with omniperf profile and then obtained roofline chart PDFs using omniperf profile --kernel-names --roof-only, then we have both the profiling data and the roofline PDFs. If we reversed the order of these two operations, we lose the PDFs and legend. Could this be fixed that the user does not have to remember this ordering detail?

Development Environment:

To Reproduce Steps to reproduce the behavior:

  1. `omniperf profile -n new --roof-only --kernel-names --device 0 -- ./exe ^--- this will create the roofline PDFs and legend
  2. `omniperf profile -n new --device 0 -- ./exe ^--- this will collect all other profiling data, but erase the PDFs collected in the previous step.

If you did the steps in the opposite order, we will have all profiling data intact.

Expected behavior If we profile and collect different things in different commands but save to the same workload, nothing should be erased by subsequent profile steps unless the subsequent step collects the same info and has to overwrite the data collected. I can see how the second command above would have assumed that all roofline data collected earlier has to be overwritten.. but could we have the PDFs regenerated in this case?

ppanchad-amd commented 3 weeks ago

Hi @gsitaram. Internal ticket has been created to investigate your issue. Thanks!

zichguan-amd commented 3 weeks ago

Hi @gsitaram, the default behaviour is to clear the workload directory if it exists during setup, see this line. If you run multiple workloads with the same name, the directory will be cleared in-between runs. This does not happen with the --roof-only flag so you should run the roofline after you have done the profiling.

gsitaram commented 2 weeks ago

It is not a good idea to ask users to remember an order of issuing commands. I can think of some possibilities. Please comment on them.