Implement/document TB debugging with tf.keras model.fit

[gthb]

Currently (in TF 2.0.0 and TF 2.1.0 and tf-nightly/tb-nightly) the /#debugger view shows this snippet for “Keras Model”:

import tensorflow as tf
from tensorflow.python import debug as tf_debug
import keras

keras.backend.set_session(
    tf_debug.TensorBoardDebugWrapperSession(tf.Session(), "ip-a-b-c-d:6064"))
# Define your keras model, called "model".
model.fit(...)

But I'm using tf.keras. So I tried the above with from tensorflow import keras instead of import keras, and that fails (in TF 2.0.0 and TF 2.1.0 and tf-nightly) with:

AttributeError: module 'tensorflow.python.keras.api._v2.keras.backend' has no attribute 'set_session'

Looks like set_session was marked compat.v1 already in Tensorflow 2.0.0: https://github.com/tensorflow/tensorflow/commit/86ac0d0a377d5b89a97c9c468a31b6506c9a7209

So I didn't find any instructions anywhere for using the Tensorboard debugger view with a tf.keras model.fit scenario in Tensorflow 2.0.0 up.

Is that not implemented at all, or the setup just not documented?

[rithiksachdev]

Yes, you are right. set_session is marked compat.v1 You can implement your code in the following two ways ->

import tensorflow as tf
from tensorflow.python import debug as tf_debug
from tensorflow import keras

tf.compat.v1.keras.backend.set_session(
    tf_debug.TensorBoardDebugWrapperSession(tf.compat.v1.Session(), "ip-a-b-c-d:6064"))
    //your code
model.fit(...)

or like ->

import tensorflow as tf
from tensorflow.python import debug as tf_debug
import keras

keras.backend.set_session(
    tf_debug.TensorBoardDebugWrapperSession(tf.compat.v1.Session(), "ip-a-b-c-d:6064"))
    //your code
model.fit(...)

I hope this solves your problem.

[rmothukuru]

@gthb, Can you please let us know if @rithiksachdev's workaround has solved your problem. Thanks!

[gthb]

Hi @rithiksachdev and @rmothukuru, nope, neither of these works. With the first approach, I just see this warning logged at the start of training:

From ...mymodule.py:274: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

(despite the fact that I am calling tf.compat.v1.keras.backend.set_session...)

... and then training starts, but the debugger never connects. Tensorboard debugger view still shows the “Debugger is waiting for Session.run() connections...” box, and sudo lsof -i :6064 shows just the tensorboard process listening on that port, but nothing connecting to it. Also tried running set_session before creating the model, as the Tensorboard debugger view says I should, but it's the same.

With the second approach (after pip install keras), the set_session call raises an exception:

  ...
  File ".../mymodule.py", line 274, in train
    keras.backend.set_session(tf_debug.TensorBoardDebugWrapperSession(tf.compat.v1.Session(), "127.0.0.1:6065"))
  File ".../lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 400, in set_session
    '`set_session` is not available '
RuntimeError: `set_session` is not available when using TensorFlow 2.0.

[1OGAG]

I'm on TF 2.1.0 and have a similar problem. I can launch TensorBoard debugger using rithiksachdev first solution. I see "Debugger is waiting for Session.run() connections..." but when I try to fit a model..nothing. I am stuck on the waiting screen. Can anyone confirm that the Tensorboard Debugger works on TF 2.1.0? --Thanks

[rantav]

I too see similar (bogus) behavior as @gthb describes, usingTensorFlow version: 2.1.0

[bazhenov]

I'm facing the same issue in TF/2.2.0. Tensorflow doesn't even try to connect to Tensorboard debugger plugin.