Visualizing profiling results for multi-GPUs

[victorapm]

I'm interested in visualizing profiling results for multi-GPU runs with MPI and check how kernels running in different GPUs line up. However, it seems that perfetto supports loading only one trace file at a time. Are there any recommendations for visualizing multiple trace files in the same window? Thank you!

[jrmadsen]

Hi @victorapm, in the past cat *.proto > combined.pftrace worked, and then it stopped working, and then recently, it started working again. I am unsure if the lack of support at times is due to an issue in the visualization or the file format but it is worth a shot. Alternatively, you will want to configure omnitrace’s perfetto backend in “system mode” and start up a perfetto daemon; the instructions can be found here

[victorapm]

@jrmadsen thanks for the reply!

The first idea didn't work. I will check with my system's admin about the second idea as it requires installing perfetto.

I was reading omnitrace's documentation and found: https://github.com/ROCm/omnitrace/blob/6bcd1d36cc0b8ee8b38154ccc8c67fd42da9a664/source/docs/installation.md?plain=1#L235-L238

So, if I build omnitrace with OMNITRACE_USE_MPI=ON and set OMNITRACE_PERFETTO_COMBINE_TRACES, would this be a third option to get the MPI tracing to work without having to build perfetto?

[jrmadsen]

I will check with my system's admin about the second idea as it requires installing perfetto.

in the section I sent you, it notes that Omnitrace release installed from the STGZ provides the perfetto installation with the command line tools (traced -> omnitrace-perfetto-traced, perfetto -> omnitrace-perfetto).

I was reading omnitrace's documentation and found

No this is effectively the same as the first option, it just automates it.

[victorapm]

Sounds good! Thanks for the support!