Feature request: re-add activation histograms

[bersbersbers]

System information.

TensorFlow version: 2.8.0-rc1

Describe the feature and the current behavior/state.

https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard says:

histogram_freq: frequency (in epochs) at which to compute activation and weight histograms for the layers of the model. If set to 0, histograms won't be computed. Validation data (or split) must be specified for histogram visualizations.

But I am fairly certain activation histograms are not written, see also https://github.com/tensorflow/tensorflow/issues/39755 and https://github.com/tensorflow/tensorflow/issues/42027.

Solutions:

1. re-implement the documented feature (it existed before)
2. un-document the un-implemented feature

Will this change the current api? How?

1 will not change the API; 2 will change the docs.

Who will benefit from this feature?

Everyone

[Cheril311]

I can try to solve this

[Cheril311]

Hey @bersbersbers as far as I know you can visualize histograms in tensorboard by using tf.summary.histogram
Can you please attach your code so that I can see if we are on the same page?

[bersbersbers]

@Cheril311 please see https://github.com/tensorflow/tensorflow/issues/42027#issuecomment-670719469. Keras writes weight histograms, but no activation histograms.

[Cheril311]

You seem to be right @bersbersbers , for now I have changed the documentation as I do know how to compute activation histograms but I am unaware on how to write them as tensorboard images

[bersbersbers]

I do know how to compute activation histograms but I am unaware on how to write them as tensorboard images

I would think activation histograms can be written the same way as the weight histograms, at least in a first iteration. After all, both should have the exact same dimensions.

[Cheril311]

I do know how to compute activation histograms but I am unaware on how to write them as tensorboard images

I would think activation histograms can be written the same way as the weight histograms, at least in a first iteration. After all, both should have the exact same dimensions.

Yup I agree with that but there is some other process done in the function _log_weight_as_image in callbacks.py and I am not sure how to do the same for activations, if you can help me maybe we can submit a pr

[bersbersbers]

I would think activation histograms can be written the same way as the weight histograms, at least in a first iteration. After all, both should have the exact same dimensions.

Yup I agree with that but there is some other process done in the function _log_weight_as_image in callbacks.py and I am not sure how to do the same for activations, if you can help me maybe we can submit a pr

To my understanding, activations (for one sample) and weights have the same shape, so you can do exactly the same thing as here:

https://github.com/keras-team/keras/blob/8015f697d0109ccd6021fe1b8f2bf5b133b259b9/keras/callbacks.py#L2570-L2580

Since you mentioned in https://github.com/keras-team/tf-keras/issues/118 that you know how to compute activation histograms (although I have yet to find a good solution myself, compare https://stackoverflow.com/q/60816678, https://gist.github.com/SiLiKhon/3965c967c3283feccc79822e6252b34c, https://stackoverflow.com/q/66779524), this should be straightforward then. Good luck!

[haifeng-jin]

Summary: The tensorboard callback doesn't print activation histograms. The solution is either change the docs or fix this bug.

[haifeng-jin]

This should have been fixed. Please reopen if it is not.

[bersbersbers]

@haifeng-jin may I ask where I can see the change? Did you update the docs or implement the functionality?

I don't see any commit on https://github.com/keras-team/keras/commits/master, and https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/TensorBoard (which has been updated for TB2.8.0) still mentions "Activation histograms".

[haifeng-jin]

@bersbersbers Do you reproduce it in TF-nightly? The commit should be sometime ago. This was from the team member who handled this. If so, you can create a Colab notebook, which can show tensorboard in it. Thanks

[bersbersbers]

@haifeng-jin I am getting errors with TensorBoard on tf-nightly (ValueError: Duplicate plugins for name projector), so I cannot test it there. The code that reproduces the issue on TF/TB2.8.0 with keras-nightly (I guess this is the important nightly part) is this:

# pip install tensorflow
# pip uninstall keras
# pip install keras-nightly
import tensorflow as tf
layer = tf.keras.layers.Input(shape=())
model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(4, input_shape=(4,)))
model.compile(optimizer="adam", loss="mse")
model.summary()
cb = tf.keras.callbacks.TensorBoard("logs", histogram_freq=1)
model.fit(x=[[1,2,3,4]], y=[[1,2,3,4]], epochs=2, callbacks=[cb])
# tensorboard --logdir logs

And then there's weight histograms, but not activation histograms, just like 1.5 years ago.

[bersbersbers]

(By the way, I'd like to reopen, but cannot.)

[LukeWood]

This is now a feature request and contributions are welcome

[bersbersbers]

Thanks. For the record, I had previously missed the fix to the documentation in 2ba6dea / https://github.com/tensorflow/tensorflow/pull/54196, hence my confusion.

[kydonian]

Suggest renaming this issue to "Feature request: re-add activation histograms"

[Shreyz-max]

@LukeWood anyone working on it? I would like to if no one else is.

[LukeWood]

@LukeWood anyone working on it? I would like to if no one else is.

Dont think so!