search

NVlabs / NVBit

220 stars 20 forks source link

Very large and wrong stream ID #85

Open mkhairy opened 2 years ago

mkhairy commented 2 years ago

Hello,

The stream id reported by the nvbit seems unusually large and looks wrong. Any help to what could be causing this issue? Is this a bug? I remember you are serializing the kernels when running with Nvbit, could that be causing an issue? The variable/structure ID that we use to report stream Id is as follows:

cuLaunchKernel_params *p = (cuLaunchKernel_params *)params;
fprintf(resultsFile, "-cuda stream id = %lu\n", (uint64_t)p->hStream);

Some stream Ids reported are like that "26321836902". It is either reporting 0 or big large numbers that do not match the stream Id used in the code. Unless hStream is a pointer to another structure in which I have to call a member called "streamId". If this case, what exactly is the right way to print stream ID?

Thanks for your help!

x-y-z commented 2 years ago

Which streamId are you referring to? Do you have a sample code?

Based on CUDA simpleStream example, when a stream is created, an opaque handle is returned. For example: https://github.com/NVIDIA/cuda-samples/blob/master/Samples/0_Introduction/simpleStreams/simpleStreams.cu#L335

You probably want to print out the stream handler pointer in your program, so that you can match it with what you see from NVBit.

christindbose commented 2 years ago

@x-y-z , thanks for your reply. We are primarily interested to build a kernel dependency graph. The motivation for this is to build a scheduler in our gpu architectural simulator that supports concurrent kernel execution.

Our approach as of now has been to first run the apps using a profiler like (nsight systems) which gives information like the kernel start/stop times and the stream id. We can build a dependency graph from this information. Then we want to correlate these kernel stream id to the ones dumped out by NVbit. Attached is a snapshot of the streamid from the simpleStream example dumped out by NVBIT vs nsight systems.

If we can directly get a dependency graph from NVBIT, that's fine as well. Let us know what you think! simplestreams_streamid

christindbose commented 2 years ago

@x-y-z , As Mahmoud mentioned, the value dumped by NVBIT is an opaque handle but somehow the nsight systems tool can infer the stream id from that handle. We suspect that the stream id is hidden at some offset within that opaque handle.

christindbose commented 2 years ago

@x-y-z If there's a better way to get the kernel dependency graph, pls let us know your thoughts.