ROCm / roctracer

ROCm Tracer Callback/Activity Library for Performance tracing AMD GPUs
https://rocm.docs.amd.com/projects/roctracer/en/latest/
Other
69 stars 30 forks source link

can we make the call to `roctracer_flush_activity` blocking? #39

Closed deven-amd closed 1 year ago

deven-amd commented 4 years ago

Currently it seems that roctracer_flush_activity is non-blocking, i.e it will return before the Activity_Callback routine has been called on the activity records being flushed out. This leads to undesired scenario on the appilcation side. Assume the application code (that uses roctracer) does something like

...
roctracer_properties_t properties{};
properties.buffer_callback_fun = ActivityCallback;
...
roctracer_open_pool(&properties)
...
...
roctracer_flush_activity();
...
aggregrate_roctracer_event_data_and_serialize_it_to_file();
...

For small testcases, ActivityCallback gets called (as a consequence of the flush) before the aggregrate_roctracer_event_data_and_serialize_it_to_file routine gets called, and things work otu just fine.

But for "real" testcases (like BERT, on which I am trying this out right now), the call to ActivityCallback seems to happen after the call to aggregrate_roctracer_event_data_and_serialize_it_to_file , resulting in some trace data getting dropped/lost.

Question is how to ensure this does not happen. One way would be to make the call to roctracer_flush_activity blocking.

deven-amd commented 4 years ago

this issue is leading to a lot of activity records getting dropped in large testcases.

lmoriche commented 1 year ago

In ROCm-5.3, roctracer_flush_activity() is a blocking call.