Open MaHaWo opened 3 months ago
Do you see meaningful information under the trace viewer tool? I'd start there rather than the overview tool.
Thank you for your answer. I do get output from the trace viewer, but I'm still confused by the warning message, since it seems to indicate that profiling didn´t work. Is there any documentation on this? is it intentional or unavoidable?
Hello! I have met the same problem while using tensorboard profiling, have you resolved this issue?
what I ultimately did was using nvidia insight directly, which at least gives you some indication of what's going on. See here: https://github.com/NVIDIA/JAX-Toolbox/blob/main/docs/profiling.md
The overview tool in tensorboard isn't really designed to work well with JAX. I'd recommend mostly ignoring it at the moment and looking at the trace viewer.
Description
I'm trying to follow the documentation on profiling and am stuck when trying to evaluate the traces with tensorboard, running the following code:
which is shown in the docs.
Now, trying to view the trace with tensorboard:
I always get the warning
No step marker observed and hence the step time is unknown. This may happen if (1) training steps are not instrumented (e.g., if you are not using Keras) or (2) the profiling duration is shorter than the step time. For (1), you need to add step instrumentation; for (2), you may try to profile longer.
No profiling data is available or shown. `The code itself produces only the following warning:
which I am not sure is relevant here.
What am I doing wrong? How could I go about troubleshooting this? thanks in advance
System info (python version, jaxlib version, accelerator, etc.)
accelerator: GPU
EDIT: correct highlighting of error messages