deepfakes / faceswap-playground

User dedicated repo for the faceswap project
306 stars 194 forks source link

CUBLAS_STATUS_ALLOC_FAILED and CUDNN_STATUS_ALLOC_FAILED #292

Closed VSBDev closed 5 years ago

VSBDev commented 5 years ago

Expected behavior

Images are converted to the output folder

Actual behavior

The following error appears: 06/14/2019 11:50:10 INFO Loading Mask from Mask_Blend plugin...

06/14/2019 11:50:10 WARNING Predicted selected, but the model was not trained with a mask. Switching to 'dfl_full'

06/14/2019 11:50:10 INFO Loading Color from Avg_Color plugin...

2019-06-14 11:50:14.027064: E tensorflow/stream_executor/cuda/cuda_blas.cc:510] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2019-06-14 11:50:14.028614: E tensorflow/stream_executor/cuda/cuda_blas.cc:510] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED 2019-06-14 11:50:17.380488: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED 2019-06-14 11:50:17.382457: E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

Other relevant information

Windows 10 Python 3.6 Installed via anaconda with setup.py Extract and training steps work fine

torzdf commented 5 years ago

Crash report.

VSBDev commented 5 years ago

Crash report attached


06/14/2019 11:50:10 MainProcess     MainThread      config          check_config_change       DEBUG    Default config has not changed
06/14/2019 11:50:10 MainProcess     MainThread      config          check_config_choices      DEBUG    Checking config choices
06/14/2019 11:50:10 MainProcess     MainThread      config          check_config_choices      DEBUG    Checked config choices
06/14/2019 11:50:10 MainProcess     MainThread      config          validate_config           DEBUG    Validated config
06/14/2019 11:50:10 MainProcess     MainThread      config          handle_config             DEBUG    Handled config
06/14/2019 11:50:10 MainProcess     MainThread      config          __init__                  DEBUG    Initialized: Config
06/14/2019 11:50:10 MainProcess     MainThread      _base           set_config                DEBUG    Config: {}
06/14/2019 11:50:10 MainProcess     MainThread      _base           __init__                  DEBUG    config: {}
06/14/2019 11:50:10 MainProcess     MainThread      _base           __init__                  DEBUG    Initialized Color
06/14/2019 11:50:10 MainProcess     MainThread      convert         load_plugins              DEBUG    Loaded plugins: {'box': <plugins.convert.mask.box_blend.Mask object at 0x000001ACF6B45128>, 'mask': <plugins.convert.mask.mask_blend.Mask object at 0x000001ACF6B45E80>, 'color': <plugins.convert.color.avg_color.Color object at 0x000001ACF6B45C18>, 'seamless': None, 'scaling': None}
06/14/2019 11:50:10 MainProcess     MainThread      convert         __init__                  DEBUG    Initialized Converter
06/14/2019 11:50:10 MainProcess     MainThread      convert         __init__                  DEBUG    Initialized Convert
06/14/2019 11:50:10 MainProcess     MainThread      convert         process                   DEBUG    Starting Conversion
06/14/2019 11:50:10 MainProcess     MainThread      convert         convert_images            DEBUG    Converting images
06/14/2019 11:50:10 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'convert_out'
06/14/2019 11:50:10 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'convert_out'
06/14/2019 11:50:10 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager getting: 'patch'
06/14/2019 11:50:10 MainProcess     MainThread      queue_manager   get_queue                 DEBUG    QueueManager got: 'patch'
06/14/2019 11:50:10 MainProcess     MainThread      convert         pool_processes            DEBUG    12
06/14/2019 11:50:10 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initializing PoolProcess: (target: 'Converter.process', processes: 12)
06/14/2019 11:50:10 MainProcess     MainThread      multithreading  set_procs                 VERBOSE  Processing 'Converter.process' in 1 processes
06/14/2019 11:50:10 MainProcess     MainThread      multithreading  __init__                  DEBUG    Initialized PoolProcess: 'Converter.process'
06/14/2019 11:50:10 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:10 MainProcess     MainThread      multithreading  start                     DEBUG    Pooling Processes: (target: 'Converter.process', args: (), kwargs: {'in_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1ac58922240>, 'out_queue': <AutoProxy[Queue] object, typeid 'Queue' at 0x1ac589223c8>})
06/14/2019 11:50:10 MainProcess     MainThread      multithreading  start                     DEBUG    Adding process 1 of 1 to mp.Pool 'Converter.process'
06/14/2019 11:50:10 MainProcess     MainThread      multithreading  start                     DEBUG    Pooled Processes: 'Converter.process'
06/14/2019 11:50:11 SpawnPoolWorker-13 MainThread      convert         process                   DEBUG    Starting convert process. (in_queue: <queue.Queue object at 0x000002468B758F60>, out_queue: <queue.Queue object at 0x000002468B758EB8>)
06/14/2019 11:50:11 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:11 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:11 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:12 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:12 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:12 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:13 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:13 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:13 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:14 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:14 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:14 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:14 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:15 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:15 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:16 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:16 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:16 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:17 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:17 MainProcess     predict_faces_0 multithreading  run                       DEBUG    Error in thread (predict_faces_0): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\n  [[{{node encoder/conv2d_9/convolution}}]]\n     [[{{node decoder_1/conv2d_8/Sigmoid}}]]
06/14/2019 11:50:17 MainProcess     load_0          pipeline        detected_faces            DEBUG    Running Detection. Phase: 'detect'
06/14/2019 11:50:17 MainProcess     MainThread      multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'tensorflow.python.framework.errors_impl.UnknownError'>, UnknownError(), <traceback object at 0x000001ACF73CAF88>)]
Traceback (most recent call last):
  File "C:\Users\VictorSB\Development\ML\faceswap\lib\cli.py", line 110, in execute_script
    process.process()
  File "C:\Users\VictorSB\Development\ML\faceswap\scripts\convert.py", line 100, in process
    self.convert_images()
  File "C:\Users\VictorSB\Development\ML\faceswap\scripts\convert.py", line 118, in convert_images
    self.check_thread_error()
  File "C:\Users\VictorSB\Development\ML\faceswap\scripts\convert.py", line 131, in check_thread_error
    thread.check_and_raise_error()
  File "C:\Users\VictorSB\Development\ML\faceswap\lib\multithreading.py", line 433, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\VictorSB\Development\ML\faceswap\lib\multithreading.py", line 390, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\VictorSB\Development\ML\faceswap\scripts\convert.py", line 532, in predict_faces
    predicted = self.predict(feed_faces)
  File "C:\Users\VictorSB\Development\ML\faceswap\scripts\convert.py", line 581, in predict
    predicted = self.predictor(feed)
  File "C:\Users\VictorSB\.conda\envs\faceswap\lib\site-packages\keras\engine\training.py", line 1169, in predict
    steps=steps)
  File "C:\Users\VictorSB\.conda\envs\faceswap\lib\site-packages\keras\engine\training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "C:\Users\VictorSB\.conda\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\VictorSB\.conda\envs\faceswap\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\VictorSB\.conda\envs\faceswap\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
    run_metadata_ptr)
  File "C:\Users\VictorSB\.conda\envs\faceswap\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
     [[{{node encoder/conv2d_9/convolution}}]]
     [[{{node decoder_1/conv2d_8/Sigmoid}}]]

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         3c80794 GUI: Show more previews for extract and convert. b305dde Limit draw transparent to swap area only. 6ac422a Extraction Improvements. 2324cee tf.__version__ bugfix. 5bf4cf5 Merge branch 'staging' of https://github.com/deepfakes/faceswap into staging
gpu_cuda:            Not Found. Check Conda packages for Conda Cuda
gpu_cudnn:           Not Found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1050 Ti with Max-Q Design
gpu_devices_active:  GPU_0
gpu_driver:          430.64
gpu_vram:            GPU_0: 4096MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.17763-SP0
os_release:          10
py_command:          C:\Users\VictorSB\Development\ML\faceswap\faceswap.py convert -i C:/Users/VictorSB/Documents/faceswap/donald trump/originals -o C:/Users/VictorSB/Documents/faceswap/donald trump/originals/converted -l 0.4 -m C:/Users/VictorSB/Documents/faceswap/faceswap_trump_lawrence_2_original -c avg-color -sc none -M predicted -w opencv -osc 100 -g 1 -L INFO -gui
py_conda_version:    conda 4.6.11
py_implementation:   CPython
py_version:          3.6.8
py_virtual_env:      True
sys_cores:           12
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 32272MB, Available: 13451MB, Used: 18820MB, Free: 13451MB

=============== Pip Packages ===============
absl-py==0.7.1
astor==0.7.1
certifi==2019.3.9
cloudpickle==1.1.1
cycler==0.10.0
cytoolz==0.9.0.1
dask==1.2.2
decorator==4.4.0
fastcluster==1.1.25
ffmpy==0.2.2
gast==0.2.2
grpcio==1.16.1
h5py==2.9.0
imageio==2.5.0
imageio-ffmpeg==0.3.0
joblib==0.13.2
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==2.2.2
mkl-fft==1.0.12
mkl-random==1.0.2
mkl-service==2.0.2
mock==3.0.5
networkx==2.3
numpy==1.16.2
nvidia-ml-py3==7.352.0
olefile==0.46
opencv-python==4.1.0.25
pathlib==1.0.1
Pillow==6.0.0
protobuf==3.8.0
psutil==5.6.2
pyparsing==2.4.0
pyreadline==2.1
python-dateutil==2.8.0
pytz==2019.1
PyWavelets==1.0.3
PyYAML==5.1
scikit-image==0.15.0
scikit-learn==0.21.2
scipy==1.2.1
six==1.12.0
tensorboard==1.13.1
tensorflow==1.13.1
tensorflow-estimator==1.13.0
termcolor==1.1.0
toolz==0.9.0
toposort==1.5
tornado==6.0.2
tqdm==4.32.1
Werkzeug==0.15.4
wincertstore==0.2

============== Conda Packages ==============
Could not get package list```
torzdf commented 5 years ago

Ok. There is a bug in "on-the-fly" conversion which i thought I'd squashed, but apparently not.

On the fly conversion is crap anyway, and you shouldn't use it. Generate an alignments file for your source video and try to convert again.

VSBDev commented 5 years ago

Never tried alignments yet but will do. For that I need to extract the alignments.json with the extract tab from which video in my case I am putting Lawrence O'donell to a trump speech, where do I extracts the json from?

Thank you

VSBDev commented 5 years ago

Alignments fix the problem indeed, thank you @torzdf