rmothukuru
commented
3 years ago @ydennisy, In order to expedite the trouble-shooting process, please provide a complete code snippet to reproduce the issue reported here. Thanks!
Open ydennisy opened 3 years ago
rmothukuru
commented
3 years ago @ydennisy, In order to expedite the trouble-shooting process, please provide a complete code snippet to reproduce the issue reported here. Thanks!
HelWireless
commented
3 years ago i meet the same error,code like this :
# Compile the model
model.compile(optimizer = keras.optimizers.Adagrad(learning_rate=0.01),
loss = keras.losses.MeanSquaredError(),
metrics = [keras.metrics.MeanAbsoluteError()],
)
train_dataset = get_dataset_from_csv("train_data.csv", shuffle=True, batch_size=265)
log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
test_dataset = get_dataset_from_csv("test_data.csv", batch_size=265)
# Fit the model with the training dtaa
model.fit(train_dataset, epochs=5, validation_data=test_dataset,callbacks=[tensorboard_callback])the error is ;
Epoch 1/5
1599/1599 [==============================] - 42s 25ms/step - loss: 0.7993 - mean_absolute_error: 0.7109 - val_loss: 0.9138 - val_mean_absolute_error: 0.7630
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-78-61a5081c4849> in <module>
10 test_dataset = get_dataset_from_csv("test_data.csv", batch_size=265)
11 # Fit the model with the training dtaa
---> 12 model.fit(train_dataset, epochs=5, validation_data=test_dataset,callbacks=[tensorboard_callback])
13
14 # _,rmse = model.evaluate(test_dataset, verbose=0)
~/anaconda3/envs/tf2.4/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1143 epoch_logs.update(val_logs)
1144
-> 1145 callbacks.on_epoch_end(epoch, epoch_logs)
1146 training_logs = epoch_logs
1147 if self.stop_training:
~/anaconda3/envs/tf2.4/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)
426 for callback in self.callbacks:
427 if getattr(callback, '_supports_tf_logs', False):
--> 428 callback.on_epoch_end(epoch, logs)
429 else:
430 if numpy_logs is None: # Only convert once.
~/anaconda3/envs/tf2.4/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)
2337
2338 if self.histogram_freq and epoch % self.histogram_freq == 0:
-> 2339 self._log_weights(epoch)
2340
2341 if self.embeddings_freq and epoch % self.embeddings_freq == 0:
~/anaconda3/envs/tf2.4/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _log_weights(self, epoch)
2395 for layer in self.model.layers:
2396 for weight in layer.weights:
-> 2397 weight_name = weight.name.replace(':', '_')
2398 summary_ops_v2.histogram(weight_name, weight, step=epoch)
2399 if self.write_images:
AttributeError: 'TrackableWeightHandler' object has no attribute 'name'
bileschi
commented
3 years ago Reached out to Keras team for advice on how to proceed
bileschi
commented
3 years ago Ran the demo in colab getting started, which uses a very similar flow and was unable to reproduce. Suspect that somehow the issue is entangled with a choice of model?
https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/get_started.ipynb
...
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
model.fit(x=x_train,
y=y_train,
epochs=5,
validation_data=(x_test, y_test),
callbacks=[tensorboard_callback])
...@HelWireless or @ydennisy , is it possible to reproduce your issue in colab and share (possibly with fake data or a minimal fake version of your model).
bileschi
commented
3 years ago Googlers, this is mirrored internally at b/177934117
as24-asingla
commented
3 years ago The issue persists when there are experimental preprocessing layers in the model.
ydennisy
commented
3 years ago This looks like the same issue as: https://github.com/tensorflow/tensorflow/issues/41244
EDIT:
Setting histogram_freq=0 stops this issue.
Is there a way to manually assign names to all layers?
DachuanZhao
commented
3 years ago This looks like the same issue as: tensorflow/tensorflow#41244
EDIT: Setting
histogram_freq=0stops this issue.Is there a way to manually assign names to all layers?
Setting histogram_freq=0 is a bad idea because you can't get tensorflow profiler in tensorboard if you do that .
It is a temporary solution .
DachuanZhao
commented
3 years ago @ydennisy, In order to expedite the trouble-shooting process, please provide a complete code snippet to reproduce the issue reported here. Thanks!
Here is a colab
nfelt
commented
3 years ago This issue arises in particular when using categorical string layers (e.g. tensorflow.keras.layers.experimental.keras.preprocessing.StringLookup). We're looking into how this can be fixed, but it's more of an issue with the way the layer is structured than with the TensorBoard callback per se so we don't have much control over how quickly it will be resolved. (Googlers, see b/169618190.)
For now my best suggestion is to avoid using that layer; it is marked as experimental still.
ydennisy
commented
3 years ago @nfelt thanks for the update!
Do you have a suggestion for a workaround for that layer which works fine?
nfelt
commented
3 years ago Unfortunately I'm not aware of any specific alternatives for this layer (I am not very familiar with the code in question myself, just ran into this when debugging). The most likely person to know is out of office for a little while so it may take some time to get this addressed.
mikedavy
commented
3 years ago @ydennisy a bit of a crude workaround to get this working is to subclass the TensorBoard Callback and explicitly check if there is a name attribute. This assumes that you do not want a histogram of those experimental layers (which is probably a fair assumption as not much learning happening there).
I fully expect there will be a more principled way to handle these experimental layers in the future but hopefully this should get you around this current issue.
class TBCallback(keras.callbacks.TensorBoard):
def _log_weights(self, epoch):
with self._train_writer.as_default():
with summary_ops_v2.always_record_summaries():
for layer in self.model.layers:
for weight in layer.weights:
if hasattr(weight, "name"):
weight_name = weight.name.replace(':', '_')
summary_ops_v2.histogram(weight_name, weight, step=epoch)
if self.write_images:
self._log_weight_as_image(weight, weight_name, epoch)
self._train_writer.flush()
donglinz
commented
3 years ago I am experiencing exactly the same issue.
DachuanZhao
commented
3 years ago tensorflow 2.5.0 raises the same error ...
jeisinge
commented
3 years ago FYI - I believe this is similar to https://github.com/tensorflow/tensorflow/issues/43834 as well.
tomalbrecht
commented
2 years ago I am experiencing the same issue with tensorflow 2.10.0
nfelt
commented
2 years ago Unfortunately https://github.com/tensorflow/tensorboard/issues/4530#issuecomment-774331534 is still the best suggestion I have (avoid problematic layers if using the keras callback). I've bumped the internal issue as well.
sagravela
commented
6 months ago Facing the same issue with Tensorflow 2.15.1
wendy2003888
commented
1 month ago Following the thread, same issue.
Diagnostics
Diagnostics output
`````` --- check: autoidentify INFO: diagnose_tensorboard.py source unavailable INFO: diagnose_tensorboard.py source unavailable --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=8, releaselevel='final', serial=0) INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=8, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='MacBook-Pro-3.lan', release='19.6.0', version='Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64', machine='x86_64') INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='MacBook-Pro-3.lan', release='19.6.0', version='Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64', machine='x86_64') INFO: sys.getwindowsversion(): N/A INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: has conda-meta: False INFO: $VIRTUAL_ENV: None INFO: $VIRTUAL_ENV: None --- check: installed_packages INFO: installed: tensorboard==2.4.0 INFO: installed: tensorboard==2.4.0 INFO: installed: tensorflow==2.4.0 INFO: installed: tensorflow==2.4.0 INFO: installed: tensorflow-estimator==2.4.0 INFO: installed: tensorflow-estimator==2.4.0 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.4.0' INFO: tensorboard.version.VERSION: '2.4.0' --- check: tensorflow_python_version INFO: tensorflow.__version__: '2.4.0' INFO: tensorflow.__version__: '2.4.0' INFO: tensorflow.__git_version__: 'v2.4.0-rc4-71-g582c8d236cb' INFO: tensorflow.__git_version__: 'v2.4.0-rc4-71-g582c8d236cb' --- check: tensorboard_binary_path INFO: which tensorboard: b'/Users/dennisyurkevich/.pyenv/versions/3.7.8/bin/tensorboard\n' INFO: which tensorboard: b'/Users/dennisyurkevich/.pyenv/versions/3.7.8/bin/tensorboard\n' --- check: addrinfos socket.has_ipv6 = True socket.AF_UNSPEC =Issue description
The keras callback is failing to write files to the folder:
Code: