How to use the profiler plugin?

NVIDIA / nccl

Optimized primitives for collective multi-GPU communication

Other

3.26k stars 826 forks source link

How to use the profiler plugin? #1484

Open jxh314 opened 1 month ago

jxh314 commented 1 month ago

Hello, I am trying to enable the profiling plugin in the latest version of NCCL, and I received the following output: nccl4:2376301:2376301 [1] NCCL INFO PROFILER/Plugin: Closing profiler plugin Example-profiler. It seems that the plugin loaded successfully, but I am not seeing any related data. Is there anything else that needs to be set, such as environment variables?

gcongiu commented 1 month ago

You need to set NCCL_PROFILE_DUMP_FILE variable to the name of the output file. The example profiler will generate a trace file for every rank using that name as base. You might also want to set NCCL_PROFILE_EVENT_MASK to the mask of events you want NCCL to provide.