Open andreykramer opened 3 years ago
Found the same with no need for a custom loop, just calling fit()
on a Keras model, like:
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir,
histogram_freq=1,
profile_batch=1)
history = model.fit(dataset_train,
epochs=epochs,
validation_data=dataset_val,
shuffle=False,
callbacks=[tensorboard_callback],
verbose=1)
The result in Tensorboard PROFILE should be a Step-time Graph with only one step, instead getting a chart with steps from 18 to 105. See screeshot below. I will see if I can reduce the code to a smaller sample that still reproduces the issue.
[CORRECTION] In the meanwhile, I have downloaded and run locally the Colab sample, and it DID reproduce the issue https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras
Using TF 2.4.1 and TB 2.4.1 installed with pip under Ubuntu 20.04 with CUDA 11.0.
The problem can be reproduced running the Colab example https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras
Steps to reproduce: 1) Open the Colab example 2) Runtime > Run all 3) After execution has completed, scroll down to either visualization of Tensorboard 4) In Tensorboard, select PROFILE (may have to find it under INACTIVE) 5) From the Step-time Graph of PROFILE, open the Run (2) drop-down menu, and select the second run in the list (which actually was the first to run) 6) The Step-time Graph is wrong, with incorrect step numbers
The difference between the two runs is in the Dataset pipeline. The first run, which shows the issue, builds the pipeline like:
ds_train = ds_train.map(normalize_img)
ds_train = ds_train.batch(128)
I have found that, if I add a cache()
operation after the map()
or the batch()
, then the Step-time Graph in Tensorboard seems correct.
I'm seeing the same. Using a profile_batch of 50,60
shows step numbers of 199,392,583,778,973,1172,1333,1554,1741
Those numbers match the profile steps (9) but the values are completely random and vary from run to run
I can also see the same inconsistency in the step number on the overview page for my model code. It is custom training loop similar to @andreykramer
I used the tf.profiler.experimental.Trace API to trace steps [48, 56) of my training loop. The trace events are correct, but the overview page graph is not. Also, if I change the number of GPUs then the step numbers on the overview page graph also change. Using TF 2.5.0 and CUDA 11.2.
I'm having the same issue. I'm also using tf.profiler.experimental to trace a script and the step numbers that appear seem really random.
I tried to run the example in https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras and had the same problem
Based on the tensorflow guide "writing a training loop from scratch" I've created a reproducible example (see bottom) to show that I can't seem to get the step number on the Step-time graph right. I adapted this example on tf.profiler.experimental.Trace to trace steps [20,29] on my training loop. The trace is correct:![imagen](https://user-images.githubusercontent.com/27460857/105739728-b71efa00-5f38-11eb-8060-ed234e27adfd.png)
But on the step-time graph in the overview page, the range of the X axis is [1,8]:![imagen](https://user-images.githubusercontent.com/27460857/105739809-cf8f1480-5f38-11eb-841d-821833828bcc.png)
It's even worse on the actual code I'm trying to get the profiler working on, where I try to trace in a similar way steps [20,29] but the resulting step-time graph looks like this:![imagen](https://user-images.githubusercontent.com/27460857/105740842-e08c5580-5f39-11eb-9c56-938279e95c9b.png)
Am I getting something wrong? Where does [1,8] range come from?
Thank you in advance.
Here's the code for the reproducible example: