search

NVIDIA / NVTX

The NVIDIA® Tools Extension SDK (NVTX) is a C-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications.
Apache License 2.0
310 stars 48 forks source link

NVTX can't annotate backward process of AI model training #101

Open JuiceLemonLemon opened 3 months ago

JuiceLemonLemon commented 3 months ago

Hello, Good afternoon!

When I annotate my training process by NVTX, then show in Nsight system, I found the forward process is annotated successfully, but the backward process is not. I don't know why.

                    with nvtx.annotate("backward path", color="yellow"):
                        loss.backward()
### Tasks
evanramos-nvidia commented 4 weeks ago

A simple explanation of what's happening: The backward pass happens in a different thread, so the NVTX ranges generated by those operations are not seen as decedents of the scoped range that encases the call to .backward().