自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED

我租了一台8个GPU、单GPU 84G显存的tesla 100，使用缺省头像生成是没问题的。如果使用自动定义头像（我自己的照片，7MB），创建虚拟角色的步骤没有问题（就是生成这个me.mp4)，但在生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED


# python general_demo.py     --human ./file/input/me.mp4     --output me.mp4     --text 各位开发者大家好，我是您的专属虚拟主播，很高兴能为您服务。

/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/librosa/core/constantq.py:1059: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  dtype=np.complex,
[01/05 02:11:00] ppgan INFO: Found /root/.cache/ppgan/GPEN-512.pdparams
W0105 02:11:01.451380 57662 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 8.0, Driver API Version: 11.5, Runtime API Version: 11.2
W0105 02:11:01.453052 57662 device_context.cc:465] device: 0, cuDNN Version: 8.1.
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_0. generator.noises.noise_0 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_1. generator.noises.noise_1 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_2. generator.noises.noise_2 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_3. generator.noises.noise_3 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_4. generator.noises.noise_4 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_5. generator.noises.noise_5 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_6. generator.noises.noise_6 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_7. generator.noises.noise_7 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_8. generator.noises.noise_8 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_9. generator.noises.noise_9 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_10. generator.noises.noise_10 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_11. generator.noises.noise_11 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_12. generator.noises.noise_12 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_13. generator.noises.noise_13 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py:1436: UserWarning: Skip loading for generator.noises.noise_14. generator.noises.noise_14 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
[2022-01-05 02:11:02,854] [    INFO] [log.py] [L57] - File /root/.paddlespeech/models/fastspeech2_csmsc-zh/fastspeech2_nosil_baker_ckpt_0.4.zip md5 checking...
[2022-01-05 02:11:03,674] [    INFO] [log.py] [L57] - File /root/.paddlespeech/models/pwgan_csmsc-zh/pwg_baker_ckpt_0.4.zip md5 checking...
encoder_type is transformer
decoder_type is transformer
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/framework/io.py:415: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  if isinstance(obj, collections.Iterable) and not isinstance(obj, (
Building prefix dict from the default dictionary ...
[2022-01-05 02:11:06] [DEBUG] [__init__.py:113] Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
[2022-01-05 02:11:06] [DEBUG] [__init__.py:132] Loading model from cache /tmp/jieba.cache
Loading model cost 0.470 seconds.
[2022-01-05 02:11:07] [DEBUG] [__init__.py:164] Loading model cost 0.470 seconds.
Prefix dict has been built successfully.
[2022-01-05 02:11:07] [DEBUG] [__init__.py:166] Prefix dict has been built successfully.
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/math_op_patch.py:251: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.int64, but right dtype is paddle.int32, the right dtype will convert to paddle.int64
  warnings.warn(
Reading video frames...
Number of frames available for inference: 300
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/librosa/util/utils.py:2099: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  np.dtype(np.float): np.complex,
/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/librosa/util/utils.py:2099: DeprecationWarning: `np.complex` is a deprecated alias for the builtin `complex`. To silence this warning, use `complex` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.complex128` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  np.dtype(np.float): np.complex,
Length of mel chunks: 157
[2022-01-05 02:11:13,369] [    INFO] - unique_endpoints {''}
[2022-01-05 02:11:13,370] [    INFO] - Found /root/.cache/paddle/hapi/weights/wav2lip_hq.pdparams
Model loaded
  0%|                                                                                                                                                                                 | 0/2 [00:00
    video = wav2lip(args.human,wavfile,args.output) #将音频合成到唇形视频
  File "/root/PaddleBoBo/PaddleTools/GAN.py", line 11, in wav2lip
    wav2lip_predictor.run(input_video, input_audio, output)
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/apps/wav2lip_predictor.py", line 257, in run
    for i, (img_batch, mel_batch, frames, coords) in enumerate(
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/apps/wav2lip_predictor.py", line 118, in datagen
    face_det_results = self.face_detect(
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/apps/wav2lip_predictor.py", line 74, in face_detect
    detector.get_detections_for_batch(
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/faceutils/face_detection/api.py", line 73, in get_detections_for_batch
    detected_faces = self.face_detector.detect_from_batch(images.copy())
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/faceutils/face_detection/detection/sfd/sfd_detector.py", line 56, in detect_from_batch
    bboxlists = batch_detect(self.face_detector, images)
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/faceutils/face_detection/detection/sfd/detect.py", line 76, in batch_detect
    olist = net(imgs)
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py", line 914, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/ppgan/faceutils/face_detection/detection/sfd/net_s3fd.py", line 133, in forward
    h = F.relu(self.conv1_1(x))
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/fluid/dygraph/layers.py", line 914, in __call__
    outputs = self.forward(*inputs, **kwargs)
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/nn/layer/conv.py", line 665, in forward
    out = F.conv._conv_nd(
  File "/root/anaconda3/envs/paddle_env/lib/python3.9/site-packages/paddle/nn/functional/conv.py", line 123, in _conv_nd
    pre_bias = getattr(_C_ops, op_type)(x, weight, *attrs)
OSError: (External) CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED.
  [Hint: 'CUDNN_STATUS_NOT_SUPPORTED'.  The functionality requested is not presently supported by cuDNN.  ] (at /paddle/paddle/fluid/platform/cudnn_desc.h:171)
  [operator < conv2d > error]

在aistudio租赁32G的机器，直接报内存不够

SystemError: (Fatal) Operator conv2d raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 38.074219GB memory on GPU 0, 3.777954GB memory has been allocated and available memory is only 27.941284GB.

Please check whether there is any other process using GPU 0.

If yes, please stop them, or start PaddlePaddle on another GPU.
If no, please decrease the batch size of your model.

(at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79) . (at /paddle/paddle/fluid/imperative/tracer.cc:221)

图像太大了，压缩自定义图片到1080p以下

------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午9:22 @.>; @.***>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9)

在aistudio租赁32G的机器，直接报内存不够

SystemError: (Fatal) Operator conv2d raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 38.074219GB memory on GPU 0, 3.777954GB memory has been allocated and available memory is only 27.941284GB.

Please check whether there is any other process using GPU 0.

If yes, please stop them, or start PaddlePaddle on another GPU.

If no, please decrease the batch size of your model.

(at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79) . (at /paddle/paddle/fluid/imperative/tracer.cc:221)

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.Message ID: @.***>

我已经压缩到1440x1080了，和第一个帖子一样，直接报CUDNN_STATUS_NOT_SUPPORTED

图像太大了，压缩自定义图片到1080p以下 … ------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午9:22 @.>; @.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9) 在aistudio租赁32G的机器，直接报内存不够 SystemError: (Fatal) Operator conv2d raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError: Out of memory error on GPU 0. Cannot allocate 38.074219GB memory on GPU 0, 3.777954GB memory has been allocated and available memory is only 27.941284GB. Please check whether there is any other process using GPU 0. If yes, please stop them, or start PaddlePaddle on another GPU. If no, please decrease the batch size of your model. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79) . (at /paddle/paddle/fluid/imperative/tracer.cc:221) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.Message ID: @.>

确认是32G显存的机器，试下长宽再压缩一半？

------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午10:02 @.>; @.**@.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9)

我已经压缩到1440x1080了，和第一个帖子一样，直接报CUDNN_STATUS_NOT_SUPPORTED

图像太大了，压缩自定义图片到1080p以下 … ------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午9:22 @.>; @.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9) 在aistudio租赁32G的机器，直接报内存不够 SystemError: (Fatal) Operator conv2d raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError: Out of memory error on GPU 0. Cannot allocate 38.074219GB memory on GPU 0, 3.777954GB memory has been allocated and available memory is only 27.941284GB. Please check whether there is any other process using GPU 0. If yes, please stop them, or start PaddlePaddle on another GPU. If no, please decrease the batch size of your model. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79) . (at /paddle/paddle/fluid/imperative/tracer.cc:221) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.Message ID: @.>

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>

确定，是至尊版，压缩到591x443终于行了，不过这解释度也忒低了吧

确认是32G显存的机器，试下长宽再压缩一半？ … ------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午10:02 @.>; @.**@.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9) 我已经压缩到1440x1080了，和第一个帖子一样，直接报CUDNN_STATUS_NOT_SUPPORTED 图像太大了，压缩自定义图片到1080p以下 … ------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午9:22 @.>; @.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9) 在aistudio租赁32G的机器，直接报内存不够 SystemError: (Fatal) Operator conv2d raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError: Out of memory error on GPU 0. Cannot allocate 38.074219GB memory on GPU 0, 3.777954GB memory has been allocated and available memory is only 27.941284GB. Please check whether there is any other process using GPU 0. If yes, please stop them, or start PaddlePaddle on another GPU. If no, please decrease the batch size of your model. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79) . (at /paddle/paddle/fluid/imperative/tracer.cc:221) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>

显卡很容易炸，但是我试过1920*1080成功的，不是很清楚为什么1440x1080为什么会炸掉

------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午10:18 @.>; @.**@.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9)

确定，是至尊版，压缩到591x443终于行了，不过这解释度也忒低了吧

确认是32G显存的机器，试下长宽再压缩一半？ … ------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午10:02 @.>; @.@.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9) 我已经压缩到1440x1080了，和第一个帖子一样，直接报CUDNN_STATUS_NOT_SUPPORTED 图像太大了，压缩自定义图片到1080p以下 … ------------------ 原始邮件 ------------------ 发件人: "JiehangXie/PaddleBoBo" @.>; 发送时间: 2022年1月6日(星期四) 上午9:22 @.>; @.>; 主题: Re: [JiehangXie/PaddleBoBo] 自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED (Issue #9) 在aistudio租赁32G的机器，直接报内存不够 SystemError: (Fatal) Operator conv2d raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError: Out of memory error on GPU 0. Cannot allocate 38.074219GB memory on GPU 0, 3.777954GB memory has been allocated and available memory is only 27.941284GB. Please check whether there is any other process using GPU 0. If yes, please stop them, or start PaddlePaddle on another GPU. If no, please decrease the batch size of your model. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:79) . (at /paddle/paddle/fluid/imperative/tracer.cc:221) — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you are subscribed to this thread.Message ID: @.> — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you commented.Message ID: @.***>

JiehangXie / PaddleBoBo

自定义头像生成demo video的时候失败了，报CUDNN_STATUS_NOT_SUPPORTED #9