search

ROCm / clr

MIT License
85 stars 35 forks source link

Enable DEBUG_CLR_GRAPH_PACKET_CAPTURE, the packet capture function can't work with kernel node modified by hipGraphExecKernelNodeSetParams #42

Closed kevin-gjm closed 5 months ago

kevin-gjm commented 6 months ago

precondition: enable DEBUG_CLR_GRAPH_PACKET_CAPTURE

steps:

  1. hipGraphCreate
  2. hipGraphAddKernelNode
  3. hipGraphInstantiate
  4. hipGraphExecKernelNodeSetParams
  5. hipGraphLaunch

  6. hipStreamSynchronize

    After function hipGraphInstantiate, the AQL packet is generated by CaptureAQLPackets which will be used in hipGraphLaunch step.

But when with hipGraphExecKernelNodeSetParams, the AQL will be modified, and the modified AQL will not executed in hipGraphLaunch.

I think this is an issue.

iassiour commented 6 months ago

Thank you @kevin-gjm for raising. I can see the issue from the steps above as the kernel argument offset in particular may become outdated after hipGraphExecKernelNodeSetParams. Would it be possible to share a small reproducer program if you have one already?

iassiour commented 5 months ago

This has been fixed in an internal PR.

iassiour commented 5 months ago

The fix has been added to develop branch https://github.com/ROCm/clr/commit/a99b163eba57947e77e583494537717a52bc3803 and will be included in a future rocm release.