ROCm / omnitrace

Omnitrace: Application Profiling, Tracing, and Analysis
https://rocm.docs.amd.com/projects/omnitrace/en/latest/
MIT License
297 stars 27 forks source link

Missing Information for some Memory Operations (host to device or device to host) #310

Closed dwchang79 closed 9 months ago

dwchang79 commented 1 year ago

Some of my omnitrace proto files do not have the complete information on all memory operations. For example, if there is a hipMemcpy at the end from device to host, it does not show that information. I am using the following database call to find all instances:

SELECT caller_slice.name AS caller_name, caller_slice.ts AS caller_ts, caller_slice.dur AS caller_dur, caller_slice.id AS caller_id, caller_slice.name AS caller_name, caller_slice.track_id AS caller_track_id, callee_slice.ts AS callee_ts, callee_slice.dur AS callee_dur, callee_slice.id AS callee_id, callee_slice.name AS callee_name, callee_slice.track_id AS callee_track_id FROM (slice AS caller_slice JOIN flow ON caller_slice.id = flow.slice_out) JOIN slice AS callee_slice ON callee_slice.id = flow.slice_in

I think this means some of the memory operations do not have a caller or callee. I have checked the corresponding rocTracer file and it does have this information. I think something is not being picked up or lost. Or maybe it is simpler and I just need a different SQL query?

I cannot attach the proto file, but have attached the roctracer file where you can see that the hipMemcpy is in there. I will take a screenshot of what I am seeing in Perfetto. Hopefully you can see in Perfetto screen capture that the very last hipMemcpy does not link anywhere while the hipMemset does.

Thank you. roctracer-61388.txt hipmemcpy hipmemset

jrmadsen commented 10 months ago

This is likely going to be solved by the changes in #317

jrmadsen commented 9 months ago

Please re-open if #317 does not fix this issue in the upcoming release