wandb / client-ng

Experimental wandb CLI and Python API - See Experimental section below.
http://wandb.com
16 stars 7 forks source link

tensorboard_sync on tf2 keras model cant serialize histograms #127

Closed raubitsj closed 4 years ago

raubitsj commented 4 years ago

2020-08-05 13:31:17.733205: I tensorflow/core/profiler/internal/gpu/device_tracer.cc:88] GpuTracer has collected 0 callback api events and 0 activity events.
60000/60000 [==============================] - 5s 84us/sample - loss: 0.2175 - accuracy: 0.9354 - val_loss: 0.1089 - val_accuracy: 0.9675
Epoch 2/5
60000/60000 [==============================] - 5s 79us/sample - loss: 0.0971 - accuracy: 0.9697 - val_loss: 0.0910 - val_accuracy: 0.9690
Epoch 3/5
60000/60000 [==============================] - 5s 83us/sample - loss: 0.0688 - accuracy: 0.9791 - val_loss: 0.0673 - val_accuracy: 0.9785
Epoch 4/5
60000/60000 [==============================] - 5s 82us/sample - loss: 0.0545 - accuracy: 0.9827 - val_loss: 0.0658 - val_accuracy: 0.9793
Epoch 5/5
6624/60000 [==>...........................] - ETA: 4s - loss: 0.0392 - accuracy: 0.9869Exception in thread Thread-30:
Traceback (most recent call last):
File "C:\Users\jeff\Anaconda3\lib\threading.py", line 926, in _bootstrap_inner
self.run()
File "C:\Users\jeff\Anaconda3\lib\threading.py", line 870, in run
self._target(*self._args, self._kwargs)
File "c:\users\jeff\client-ng\wandb\internal\tb_watcher.py", line 239, in _thread_body
self._save_row(item)
File "c:\users\jeff\client-ng\wandb\internal\tb_watcher.py", line 259, in _save_row
data[k] = json.loads(json_dumps_safer_history(v))
File "c:\users\jeff\client-ng\wandb\util.py", line 543, in json_dumps_safer_history
return json.dumps(obj, cls=WandBHistoryJSONEncoder,
kwargs)
File "C:\Users\jeff\Anaconda3\lib\json__init__.py", line 238, in dumps
**kw).encode(obj)
File "C:\Users\jeff\Anaconda3\lib\json\encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "C:\Users\jeff\Anaconda3\lib\json\encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "c:\users\jeff\client-ng\wandb\util.py", line 514, in default
return json.JSONEncoder.default(self, obj)
File "C:\Users\jeff\Anaconda3\lib\json\encoder.py", line 179, in default
raise TypeError(f'Object of type {o.class.name} '
TypeError: Object of type Histogram is not JSON serializable

60000/60000 [==============================] - 5s 84us/sample - loss: 0.0437 - accuracy: 0.9859 - val_loss: 0.0649 - val_accuracy: 0.9813
wandb:
wandb: Find user logs for this run at: .\wandb\runs\run-20200805_133113-5w65bqwh\logs\debug-20200805_133113-5w65bqwh.log
wandb: Find internal logs for this run at: .\wandb\runs\run-20200805_133113-5w65bqwh\logs\debug-internal-20200805_133113-5w65bqwh.log
wandb: Synced noble-morning-81: https://app.wandb.ai/jeffr/examples-tensorboard-tf2-kerasfit/runs/5w65bqwh

raubitsj commented 4 years ago

reproduce: examples\tensorboard-tf2-kerasfit python train.py

raubitsj commented 4 years ago

Fixed