Closed ayaka14732 closed 1 year ago
I think the TensorFlow profiler is no longer dumping trace.json
. I'll investigate and hopefully figure out what happened.
I'm hitting this problem as well. I'm not sure if this is helpful, but when I look in the "invalid trace folder" I see only a single .pb
file (and no trace.json
)
...
ValueError: Invalid trace folder: /tmp/jax-trace/plugins/profile/2022_10_31_09_53_24
$ cd /tmp/jax-trace/plugins/profile/2022_10_31_09_53_24
$ ls
my_computer_name.xplane.pb
I'm experiencing the same problem. Have there been any advances on this?
We are currently working on automatically parsing the xplane.pb file and uploading it to Perfetto. Will update this thread when it's done! We have a Thanksgiving holiday next week so hopefully we'll have something to show the week after (cc: @pschuh)
+1
+1
Any news?
Apologies for the lack of updates! Both Parker and I have been on largely nonoverlapping vacation for the last month or so.
I'm back next week so will hopefully have something for you then.
Running into the same error. With perfetto flags set to False, the program executes. A single file is created ...xplane.pb
However, Tensorboard does not recognize the created file.
Running
tensorboard --inspect --event_file=plugins/profile/2023_01_22_17_22_49/taurusi8017.xplane.pb
yields the following output, which seems like the files are empty
======================================================================
Processing event files... (this can take a few minutes)
======================================================================
These tags are in plugins/profile/2023_01_22_17_22_49/taurusi8017.xplane.pb:
audio -
histograms -
images -
scalars -
tensor -
======================================================================
Event statistics for plugins/profile/2023_01_22_17_22_49/taurusi8017.xplane.pb:
audio -
graph -
histograms -
images -
scalars -
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -
======================================================================
Using
jax 0.4.1
jaxlib 0.4.1+cuda11.cudnn86
tensorboard 2.9.1
tensorboard-data-server 0.6.1
tensorboard-plugin-profile 2.8.0
tensorboard-plugin-wit 1.8.1
tensorflow 2.9.1
A quick update: @pschuh has made progress on reviving the old code that generated the trace.json.gz
that was uploaded to Perfetto. Once that lands, and we cut a jaxlib release, Perfetto should work again!
A single file is created ...xplane.pb However, Tensorboard does not recognize the created file.
This seems right, as of now the profiler will only generate the xplane.pb. However, Tensorboard should recognize it. Did you try pointing the --logdir
to the logdir in the profiler?
It's back! https://github.com/tensorflow/tensorflow/commit/b1dfc9285409bd9cb07f4598737450773daec573
We should be cutting a release soon so I will update the thread when that's out
Hi there, I was wondering if there are any updates on this front?
Ah sorry forgot to update the thread. I think it should work with the latest Jax.
I also met this problem and found it can be resolved by compile and reinstall jaxlib with tensorflow>=2.12
Description
Following https://jax.readthedocs.io/en/latest/profiling.html:
Output:
What jax/jaxlib version are you using?
jax v0.3.23, jaxlib v0.3.22, tensorflow v2.11.0rc1 (compatible with jaxlib)
Which accelerator(s) are you using?
TPU v4-16
Additional system info
Python 3.10.8, Linux 5.8.0-1035-gcp
NVIDIA GPU info
No response