alihassanijr
commented
2 years ago Hello and thank you for your interest. It really depends on what you mean by debugging.
- When it's a compilation issue, you may find using
setup.pybetter as the logs are a bit more concise. - If you want to check your implementation, you probably have to either come up with a pure python/pytorch implementation, and compare outputs of the two. That's what we typically do, which is send random tensors with different shapes through the CUDA version and the "torch version" with the same weights, compute the outputs, check if they
allclose, and then do the same for the backward pass. - If you're sure of your forward pass being correct, gradcheck is probably a better way to check if your backward pass kernel is correct.
- It's also not a bad idea to have assertions in place, even in the kernel when debugging, but I'd recommend leaving in as few assertions in the device code (the kernel) as possible and do assertions mostly before the kernel call.
If you're getting into optimization, you may find pytorch's profiler useful for measuring latency, and you'd probably need NVIDIA Nsight to profile in more detail.
I hope you find these useful, but if you need more details, please let us know.
m1710545173
Hello author, I want to change nattenqkrpb_cuda_forward_kernel to achieve the desired functions, but I don't know much about CUDA Programming and I don't know how to debug CUDA kernel. The programming tools i am using is the visual studio and the libtorch on Windows 10. Although i can debug some part of .cu file, i can't debug the cuda kernel. So, i want to know what tools and methods do you use to debug cuda programming? Please give me some suggestions!