deepfakes / faceswap-playground

User dedicated repo for the faceswap project
306 stars 194 forks source link

Unable to train (critical error after TensorBoard Logging) #269

Closed mmmikhailov closed 5 years ago

mmmikhailov commented 5 years ago

Expected behavior

My 1st try to faceswap. Training process crash.

Actual behavior

Loading... 03/14/2019 10:28:19 INFO Log level set to: INFO Using TensorFlow backend. 03/14/2019 10:28:21 INFO Model A Directory: C:\faceswap\Photo\Person1 03/14/2019 10:28:21 INFO Model B Directory: C:\faceswap\Photo\Person2 03/14/2019 10:28:21 INFO Training data directory: C:\faceswap\models 03/14/2019 10:28:21 INFO ===================================================================== 03/14/2019 10:28:21 INFO - Using live preview - 03/14/2019 10:28:21 INFO - Press 'ENTER' on the preview window to save and quit - 03/14/2019 10:28:21 INFO - Press 'S' on the preview window to save model weights immediately - 03/14/2019 10:28:21 INFO ===================================================================== 03/14/2019 10:28:22 INFO Loading data, this may take a while... 03/14/2019 10:28:22 INFO Loading Model from Original plugin... 03/14/2019 10:28:24 INFO Loading config: 'C:\faceswap\config\train.ini' 03/14/2019 10:28:24 WARNING No existing state file found. Generating. 03/14/2019 10:28:25 INFO Creating new 'original' model in folder: 'C:\faceswap\models' 03/14/2019 10:28:25 INFO Loading Trainer from Original plugin... 03/14/2019 10:28:25 ERROR 'NoneType' object has no attribute 'astype' Traceback (most recent call last): File "C:\faceswap\lib\multithreading.py", line 261, in _runner target(*args, *kwargs) File "C:\faceswap\lib\training_data.py", line 105, in load_batches imgs = self.process_face(img_path, side, is_timelapse) File "C:\faceswap\lib\training_data.py", line 155, in process_face image = self.processing.color_adjust(image) File "C:\faceswap\lib\training_data.py", line 226, in color_adjust return img.astype('float32') / 255.0 AttributeError: 'NoneType' object has no attribute 'astype' 03/14/2019 10:28:27 INFO Enabled TensorBoard Logging 03/14/2019 10:28:34 CRITICAL Error caught! Exiting... 03/14/2019 10:28:34 ERROR Caught exception in thread: 'training_0' 03/14/2019 10:28:36 ERROR Got Exception on main handler: Traceback (most recent call last): File "C:\faceswap\lib\cli.py", line 107, in execute_script process.process() File "C:\faceswap\scripts\train.py", line 101, in process self.end_thread(thread, err) File "C:\faceswap\scripts\train.py", line 126, in end_thread thread.join() File "C:\faceswap\lib\multithreading.py", line 443, in join raise thread.err[1].with_traceback(thread.err[2]) File "C:\faceswap\lib\multithreading.py", line 381, in run self._target(self._args, **self._kwargs) File "C:\faceswap\scripts\train.py", line 152, in training raise err File "C:\faceswap\scripts\train.py", line 142, in training self.run_training_cycle(model, trainer) File "C:\faceswap\scripts\train.py", line 214, in run_training_cycle trainer.train_one_step(viewer, timelapse) File "C:\faceswap\plugins\train\trainer_base.py", line 139, in train_one_step loss[side] = batcher.train_one_batch(do_preview) File "C:\faceswap\plugins\train\trainer_base.py", line 213, in train_one_batch batch = self.get_next(do_preview) File "C:\faceswap\plugins\train\trainer_base.py", line 221, in get_next batch = next(self.feed) StopIteration 03/14/2019 10:28:36 CRITICAL An unexpected crash has occurred. Crash report written to C:\faceswap\crash_report.2019.03.14.102834486501.log. Please verify you are running the latest version of faceswap before reporting Process exited.

Steps to reproduce

Simple training session via GUI.

Other relevant information

============ System Information ============ encoding: cp1251 git_branch: master git_commits: 7ae44c7 fix on gaussian sharpen gpu_cuda: 10.0 gpu_cudnn: 7.4.1 gpu_devices: GPU_0: GeForce GTX 1070, GPU_1: GeForce GTX 1070 gpu_driver: 419.35 gpu_vram: GPU_0: 8192MB, GPU_1: 8192MB os_machine: AMD64 os_platform: Windows-10-10.0.17134-SP0 os_release: 10 py_command: C:\faceswap/faceswap.py gui py_conda_version: conda 4.5.12 py_implementation: CPython py_version: 3.6.8 py_virtual_env: True sys_cores: 6 sys_processor: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel sys_ram: Total: 16317MB, Available: 10754MB, Used: 5563MB, Free: 10754MB

=============== Pip Packages =============== absl-py==0.7.0 astor==0.7.1 certifi==2019.3.9 Click==7.0 cloudpickle==0.8.0 cmake==3.13.3 cycler==0.10.0 cytoolz==0.9.0.1 dask==1.1.4 decorator==4.3.2 dlib==19.16.99 face-recognition==1.2.3 face-recognition-models==0.3.0 ffmpy==0.2.2 gast==0.2.2 grpcio==1.16.1 h5py==2.9.0 imageio==2.5.0 Keras==2.2.4 Keras-Applications==1.0.7 Keras-Preprocessing==1.0.9 kiwisolver==1.0.1 Markdown==3.0.1 matplotlib==2.2.2 mkl-fft==1.0.10 mkl-random==1.0.2 mock==2.0.0 networkx==2.2 numpy==1.15.4 nvidia-ml-py3==7.352.0 olefile==0.46 opencv-python==4.0.0.21 pathlib==1.0.1 pbr==5.1.3 Pillow==5.4.1 protobuf==3.6.1 psutil==5.6.1 pyparsing==2.3.1 pyreadline==2.1 python-dateutil==2.8.0 pytz==2018.9 PyWavelets==1.0.2 PyYAML==3.13 scikit-image==0.14.2 scikit-learn==0.20.3 scipy==1.2.1 six==1.12.0 tensorboard==1.12.2 tensorflow==1.12.0 tensorflow-estimator==1.13.0 termcolor==1.1.0 toolz==0.9.0 tornado==6.0.1 tqdm==4.31.1 Werkzeug==0.14.1 wincertstore==0.2

============== Conda Packages ==============

packages in environment at C:\Users\User\MiniConda3\envs\faceswap:

#

Name Version Build Channel

_tflow_select 2.1.0 gpu
absl-py 0.7.0 py36_0
astor 0.7.1 py36_0
blas 1.0 mkl
ca-certificates 2019.1.23 0
certifi 2019.3.9 py36_0
Click 7.0 cloudpickle 0.8.0 py36_0
cmake 3.13.3 cudatoolkit 9.0 1
cudnn 7.3.1 cuda9.0_0
cycler 0.10.0 py36h009560c_0
cytoolz 0.9.0.1 py36hfa6e2cd_1
dask-core 1.1.4 py_0
decorator 4.3.2 py36_0
dlib 19.16.99 face-recognition 1.2.3 face-recognition-models 0.3.0 ffmpeg 4.1 h6538335_1002 conda-forge ffmpy 0.2.2 freetype 2.9.1 ha9979f8_1
gast 0.2.2 py36_0
grpcio 1.16.1 py36h351948d_1
h5py 2.9.0 py36h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
imageio 2.5.0 py36_0
intel-openmp 2019.1 144
jpeg 9b hb83a4c4_2
keras 2.2.4 0
keras-applications 1.0.7 py_0
keras-base 2.2.4 py36_0
keras-preprocessing 1.0.9 py_0
kiwisolver 1.0.1 py36h6538335_0
libmklml 2019.0.3 0
libpng 1.6.36 h2a8f88b_0
libprotobuf 3.6.1 h7bd577a_0
libtiff 4.0.10 hb898794_2
markdown 3.0.1 py36_0
matplotlib 2.2.2 py36had4c4a9_2
mkl 2019.1 144
mkl_fft 1.0.10 py36h14836fe_0
mkl_random 1.0.2 py36h343c172_0
mock 2.0.0 py36h9086845_0
networkx 2.2 py36_1
numpy 1.15.4 py36h19fb1c0_0
numpy-base 1.15.4 py36hc3f5095_0
nvidia-ml-py3 7.352.0 olefile 0.46 py36_0
opencv-python 4.0.0.21 openssl 1.1.1b he774522_1
pathlib 1.0.1 py36_1
pbr 5.1.3 py_0
pillow 5.4.1 py36hdc69c19_0
pip 19.0.3 py36_0
protobuf 3.6.1 py36h33f27b4_0
psutil 5.6.1 py36he774522_0
pyparsing 2.3.1 py36_0
pyqt 5.9.2 py36h6538335_2
pyreadline 2.1 py36_1
python 3.6.8 h9f7ef89_7
python-dateutil 2.8.0 py36_0
pytz 2018.9 py36_0
pywavelets 1.0.2 py36h8c2d366_0
pyyaml 3.13 py36hfa6e2cd_0
qt 5.9.7 vc14h73c81de_0
scikit-image 0.14.2 py36ha925a31_0
scikit-learn 0.20.3 py36h343c172_0
scipy 1.2.1 py36h29ff71c_0
setuptools 40.8.0 py36_0
sip 4.19.8 py36h6538335_0
six 1.12.0 py36_0
sqlite 3.27.2 he774522_0
tensorboard 1.12.2 py36h33f27b4_0
tensorflow 1.12.0 gpu_py36ha5f9131_0
tensorflow-base 1.12.0 gpu_py36h6e53903_0
tensorflow-estimator 1.13.0 py_0
tensorflow-gpu 1.12.0 h0d30ee6_0
termcolor 1.1.0 py36_1
tk 8.6.8 hfa6e2cd_0
toolz 0.9.0 py36_0
tornado 6.0.1 py36he774522_0
tqdm 4.31.1 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_0
werkzeug 0.14.1 py36_0
wheel 0.33.1 py36_0
wincertstore 0.2 py36h7fe50ca_0
xz 5.2.4 h2fa13f4_4
yaml 0.1.7 hc54c509_2
zlib 1.2.11 h62dcd97_3
zstd 1.3.7 h508b16e_0

Kirin-kun commented 5 years ago

Check your datasets to see if there are no faces with a different size than the others (128x128 instead of 256x256 for example).

I had similar crashes at the batch building step because of that. All faces, in both A and B, must be the same size.

bryanlyon commented 5 years ago

Closing for non-response. If you're still experiencing this issue, please comment here.