Graph & Analysis not working

Kirin-kun commented 5 years ago

I deployed a new install on a brand new machine (Redhat 7.2) and it mostly works, but the graph and analysis in the gui don't seem to work.

The graph doesn't show up and when I'm trying to load the current state.json, It errors:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/tkinter/__init__.py", line 1702, in __call__
    return self.func(*args)
  File "/app/deep/faceswap/lib/gui/display_analysis.py", line 78, in load_session
    self.set_session_summary(msg)
  File "/app/deep/faceswap/lib/gui/display_analysis.py", line 108, in set_session_summary
    self.summary = self.session.full_summary
  File "/app/deep/faceswap/lib/gui/stats.py", line 129, in full_summary
    return self.summary.compile_stats()
  File "/app/deep/faceswap/lib/gui/stats.py", line 278, in compile_stats
    compiled_stats = self.sessions_stats
  File "/app/deep/faceswap/lib/gui/stats.py", line 262, in sessions_stats
    for sess_idx, ts_data in self.time_stats.items():
  File "/app/deep/faceswap/lib/gui/stats.py", line 251, in time_stats
    ts_data = self.session.tb_logs.get_timestamps()
  File "/app/deep/faceswap/lib/gui/stats.py", line 91, in get_timestamps
    for event in tf.train.summary_iterator(logfile)]
  File "/app/deep/faceswap/lib/gui/stats.py", line 90, in <listcomp>
    timestamps = [event.wall_time
  File "/app/deep/cface/lib/python3.6/site-packages/tensorflow/python/summary/summary_iterator.py", line 68, in summary_iterator
    for r in tf_record.tf_record_iterator(path):
  File "/app/deep/cface/lib/python3.6/site-packages/tensorflow/python/lib/io/tf_record.py", line 181, in tf_record_iterator
    reader.GetNext()
  File "/app/deep/cface/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 489, in GetNext
    return _pywrap_tensorflow_internal.PyRecordReader_GetNext(self)
tensorflow.python.framework.errors_impl.DataLossError: truncated record at 285062

# pip freeze
absl-py==0.7.1
astor==0.7.1
Click==7.0
cmake==3.13.3
cycler==0.10.0
decorator==4.4.0
dlib==19.17.0
face-recognition==1.2.3
face-recognition-models==0.3.0
ffmpy==0.2.2
gast==0.2.2
grpcio==1.20.0
h5py==2.9.0
imageio==2.5.0
Keras==2.2.4
Keras-Applications==1.0.7
Keras-Preprocessing==1.0.9
kiwisolver==1.0.1
Markdown==3.1
matplotlib==2.2.2
mock==2.0.0
networkx==2.3
numpy==1.16.2
nvidia-ml-py3==7.352.0
opencv-python==4.1.0.25
pathlib==1.0.1
pbr==5.1.3
Pillow==6.0.0
protobuf==3.7.1
psutil==5.6.1
pyparsing==2.4.0
python-dateutil==2.8.0
pytz==2019.1
PyWavelets==1.0.3
PyYAML==5.1
scikit-image==0.15.0
scikit-learn==0.20.3
scipy==1.2.1
six==1.12.0
tensorboard==1.13.1
tensorflow-estimator==1.13.0
tensorflow-gpu==1.13.1
termcolor==1.1.0
toposort==1.5
tqdm==4.31.1
Werkzeug==0.15.2

Using CUDA 10.0.

Maybe linked to tensorflow-1.13.1 which changed something and it's not compatible anymore?

It's not blocking (I can live without analysis and graph), but I'll just leave this issue open.

torzdf commented 5 years ago

Yeah, Tensorflow 1.13 doesn't work with the tensorboard callback, which means we can't log any of the loss data. This is potentially a Keras issue (It hasn't been updated since tf 1.13 came out).

We currently recommend TF 1.12 as our main supported version (it's also quicker), but will bump if/when this gets fixed.

torzdf commented 5 years ago

TF 1.13.1 is now fully supported.

deepfakes / faceswap-playground

Graph & Analysis not working #278