ExternalError: CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED.

[BillFei]

/content/video-subtitle-remover /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google.cloud'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2349: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google.logging'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google.iam'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('sphinxcontrib'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) 请输入视频文件路径: /content/video-subtitle-remover/test/test.mp4 Running Verify Fluid Program ... Your Paddle Fluid works well on SINGLE GPU or CPU. Your Paddle Fluid works well on MUTIPLE GPU or CPU. Your Paddle Fluid is installed successfully! Let's start deep Learning with Paddle Fluid now 使用GPU进行加速 字幕查找: 0% 0/180 [00:00<?, ?f/s]Traceback (most recent call last): File "/content/video-subtitle-remover/./backend/main.py", line 229, in sd.run() File "/content/video-subtitle-remover/./backend/main.py", line 156, in run sub_list = self.sub_detector.find_subtitle_frame_no() File "/content/video-subtitle-remover/./backend/main.py", line 77, in find_subtitle_frame_no dt_boxes, elapse = self.detect_subtitle(frame) File "/content/video-subtitle-remover/./backend/main.py", line 38, in detect_subtitle dt_boxes, elapse = self.text_detector(img) File "/content/video-subtitle-remover/backend/tools/infer/predict_det.py", line 216, in call self.predictor.run() OSError: In user code:

File "tools/export_model.py", line 288, in <module>
  main()
File "tools/export_model.py", line 284, in main
  model, arch_config, save_path, logger, input_shape=input_shape)
File "tools/export_model.py", line 197, in export_single_model
  paddle.jit.save(model, save_path)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py", line 629, in wrapper
  func(layer, path, input_spec, **configs)
File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
  return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py", line 857, in save
  inner_input_spec, with_hook=with_hook)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 528, in concrete_program_specify_input_spec
  *desired_input_spec, with_hook=with_hook)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 436, in get_concrete_program
  concrete_program, partial_program_layer = self._program_cache[cache_key]
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 801, in __getitem__
  self._caches[item_id] = self._build_once(item)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 790, in _build_once
  **cache_key.kwargs)
File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
  return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 733, in from_func_spec
  outputs = static_func(*inputs)
File "/tmp/tmpjvb2gofp.py", line 28, in forward
  false_fn_1, (x,), (x,), (x,))
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
  out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
  return true_fn(*true_args) if pred else false_fn(*false_args)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 86, in forward
  x = self.backbone(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/tmp/tmpsr4zekry.py", line 45, in forward
  for_loop_condition_0, for_loop_body_0, [__for_loop_var_index_0, i, x])
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 45, in convert_while_loop
  loop_vars = _run_py_while(cond, body, loop_vars)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 59, in _run_py_while
  loop_vars = body(*loop_vars)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 252, in forward
  x = stage(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 172, in forward
  x = self.blocks(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/container.py", line 98, in forward
  input = layer(input)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 125, in forward
  x = self.att(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 76, in forward
  x = self.conv(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/nn/layer/conv.py", line 678, in forward
  use_cudnn=self._use_cudnn)
File "/usr/local/lib/python3.7/dist-packages/paddle/nn/functional/conv.py", line 169, in _conv_nd
  type=op_type, inputs=inputs, outputs=outputs, attrs=attrs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layer_helper.py", line 44, in append_op
  return self.main_program.current_block().append_op(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 3621, in append_op
  attrs=kwargs.get("attrs", None))
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 2635, in __init__
  for frame in traceback.extract_stack():

ExternalError: CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED. 
  [Hint: Please search for the error code(9) on website (https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnStatus_t) to get Nvidia's official solution and advice about CUDNN Error.] (at /paddle/paddle/fluid/operators/fused/conv_fusion_op.cu:519)
  [operator < conv2d_fusion > error]

[BillFei]

Follow the instruction from the readme, there is no error until run "python ./backend/main.py". After input the path, it give a error as above. Is there any solution?

[YaoFANGUK]

the minimum requirement for python is 3.8

[BillFei]

In the error message, it use /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py to load paddle and run video-subtitle-remover application. But when it use GPU to run paddle model, it use /usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py. Is there any suggestion?

[YaoFANGUK]

use python 3.8 please

[YaoFANGUK]

by the way, it cannot run on colab. Do not waste your time

[BillFei]

Thanks. It can only run in the EC2?

[YaoFANGUK]

It can run on PC or cloud computer as long as you have NVdia GPU

[BillFei]

python ./backend/main.py /data/anaconda3/envs/watermark/lib/python3.8/site-packages/setuptools/sandbox.py:13: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html import pkg_resources /data/anaconda3/envs/watermark/lib/python3.8/site-packages/pkg_resources/init.py:2871: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /data/anaconda3/envs/watermark/lib/python3.8/site-packages/pkg_resources/init.py:2871: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) Please input video file path: /data/workspace/watermark/video-subtitle-remover/test/test.mp4 Running Verify Fluid Program ... python: relocation error: /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so: symbol runGemmShortApi, version libcublasLt.so.11 not defined in file libcublasLt.so.11 with link time reference

[BillFei]

My nvidia-smi as below:

[BillFei]

@YaoFANGUK have you got these error message before?

[2922832737C]

Open Video Success：/home/will/video-subtitle-remover-main/test/test_full.mp4 SubtitleArea：(597,720,329,937) Running Verify Fluid Program ... Your Paddle Fluid works well on SINGLE GPU or CPU. Your Paddle Fluid works well on MUTIPLE GPU or CPU. Your Paddle Fluid is installed successfully! Let's start deep Learning with Paddle Fluid now use GPU for acceleration [Processing] start finding subtitles... [Finished] Finished finding subtitles... [Processing] start removing subtitles... Exception in thread Thread-1: Traceback (most recent call last): File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/threading.py", line 870, in run self._target(*self._args, self._kwargs) File "gui.py", line 274, in task self.sr.run() File "/home/will/video-subtitle-remover-main/backend/main.py", line 176, in run frame = self.inpaint_frame(frame, masks) File "/home/will/video-subtitle-remover-main/backend/main.py", line 204, in inpaint_frame inpainted_frame = self.inpaint(inpainted_frame, mask) File "/home/will/video-subtitle-remover-main/backend/main.py", line 197, in inpaint img_inpainted = inpaint_img_with_lama(img, mask, config.LAMA_CONFIG, config.LAMA_MODEL_PATH, device=config.device) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama_inpaint.py", line 70, in inpaint_img_with_lama batch = model(batch) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/trainers/default.py", line 70, in forward batch['predicted_image'] = self.generator(masked_img) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/modules/ffc.py", line 366, in forward return self.model(input) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/container.py", line 217, in forward input = module(input) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/modules/ffc.py", line 284, in forward x_l, x_g = self.conv1((x_l, x_g)) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/modules/ffc.py", line 251, in forward x_l, x_g = self.ffc(x) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/modules/ffc.py", line 222, in forward out_xg = self.convl2g(x_l) l2g_gate + self.convg2g(x_g) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/modules/ffc.py", line 145, in forward output = self.fu(x) File "/home/will/miniconda3/envs/videoEnv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/will/video-subtitle-remover-main/backend/inpaint/lama/saicinpainting/training/modules/ffc.py", line 85, in forward ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm) RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR 有没有人遇到这个问题,执行一半了然后说untimeError: cuFFT error: CUFFT_INTERNAL_ERROR

[chenkaiC4]

@2922832737C 你是4090 显卡吗？如果是的话，那跟我的问题是一样的。cuda 11.7 太早起，无法支持 4090下运行 pytorch

[YaoFANGUK]

/content/video-subtitle-remover /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:121: DeprecationWarning: pkg_resources is deprecated as an API warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google.cloud'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2349: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(parent) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google.logging'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('google.iam'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('mpl_toolkits'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) /usr/local/lib/python3.10/dist-packages/pkg_resources/init.py:2870: DeprecationWarning: Deprecated call to pkg_resources.declare_namespace('sphinxcontrib'). Implementing implicit namespace packages (as specified in PEP 420) is preferred to pkg_resources.declare_namespace. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages declare_namespace(pkg) 请输入视频文件路径: /content/video-subtitle-remover/test/test.mp4 Running Verify Fluid Program ... Your Paddle Fluid works well on SINGLE GPU or CPU. Your Paddle Fluid works well on MUTIPLE GPU or CPU. Your Paddle Fluid is installed successfully! Let's start deep Learning with Paddle Fluid now 使用GPU进行加速 字幕查找: 0% 0/180 [00:00<?, ?f/s]Traceback (most recent call last): File "/content/video-subtitle-remover/./backend/main.py", line 229, in sd.run() File "/content/video-subtitle-remover/./backend/main.py", line 156, in run sub_list = self.sub_detector.find_subtitle_frame_no() File "/content/video-subtitle-remover/./backend/main.py", line 77, in find_subtitle_frame_no dt_boxes, elapse = self.detect_subtitle(frame) File "/content/video-subtitle-remover/./backend/main.py", line 38, in detect_subtitle dt_boxes, elapse = self.text_detector(img) File "/content/video-subtitle-remover/backend/tools/infer/predict_det.py", line 216, in call self.predictor.run() OSError: In user code:

File "tools/export_model.py", line 288, in <module>
  main()
File "tools/export_model.py", line 284, in main
  model, arch_config, save_path, logger, input_shape=input_shape)
File "tools/export_model.py", line 197, in export_single_model
  paddle.jit.save(model, save_path)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py", line 629, in wrapper
  func(layer, path, input_spec, **configs)
File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
  return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py", line 857, in save
  inner_input_spec, with_hook=with_hook)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 528, in concrete_program_specify_input_spec
  *desired_input_spec, with_hook=with_hook)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 436, in get_concrete_program
  concrete_program, partial_program_layer = self._program_cache[cache_key]
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 801, in __getitem__
  self._caches[item_id] = self._build_once(item)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 790, in _build_once
  **cache_key.kwargs)
File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun
  return caller(func, *(extras + args), **kw)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
  return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 733, in from_func_spec
  outputs = static_func(*inputs)
File "/tmp/tmpjvb2gofp.py", line 28, in forward
  false_fn_1, (x,), (x,), (x,))
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
  out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
  return true_fn(*true_args) if pred else false_fn(*false_args)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 86, in forward
  x = self.backbone(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/tmp/tmpsr4zekry.py", line 45, in forward
  for_loop_condition_0, for_loop_body_0, [__for_loop_var_index_0, i, x])
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 45, in convert_while_loop
  loop_vars = _run_py_while(cond, body, loop_vars)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 59, in _run_py_while
  loop_vars = body(*loop_vars)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 252, in forward
  x = stage(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 172, in forward
  x = self.blocks(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/container.py", line 98, in forward
  input = layer(input)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 125, in forward
  x = self.att(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/workspace/huangjun12/Projects/PP-OCRv4/pr/check_acc/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 76, in forward
  x = self.conv(x)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/nn/layer/conv.py", line 678, in forward
  use_cudnn=self._use_cudnn)
File "/usr/local/lib/python3.7/dist-packages/paddle/nn/functional/conv.py", line 169, in _conv_nd
  type=op_type, inputs=inputs, outputs=outputs, attrs=attrs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layer_helper.py", line 44, in append_op
  return self.main_program.current_block().append_op(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 3621, in append_op
  attrs=kwargs.get("attrs", None))
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 2635, in __init__
  for frame in traceback.extract_stack():

ExternalError: CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED. 
  [Hint: Please search for the error code(9) on website (https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnStatus_t) to get Nvidia's official solution and advice about CUDNN Error.] (at /paddle/paddle/fluid/operators/fused/conv_fusion_op.cu:519)
  [operator < conv2d_fusion > error]

这个报错是因为 你没有按照paddlepaddle的要求安装对应版本的cudnn

CUDA 工具包 10.2 配合 cuDNN v7.6.5, 如需使用 PaddleTensorRT 推理，需配合 TensorRT7.0.0.11

CUDA 工具包 11.2 配合 cuDNN v8.2.1, 如需使用 PaddleTensorRT 推理，需配合 TensorRT8.0.3.4

CUDA 工具包 11.6 配合 cuDNN v8.4.0, 如需使用 PaddleTensorRT 推理，需配合 TensorRT8.4.0.6

CUDA 工具包 11.7 配合 cuDNN v8.4.1, 如需使用 PaddleTensorRT 推理，需配合 TensorRT8.4.2.4

CUDA 工具包 11.8 配合 cuDNN v8.6.0, 如需使用 PaddleTensorRT 推理，需配合 TensorRT8.5.1.7

CUDA 工具包 12.0 配合 cuDNN v8.9.1, 如需使用 PaddleTensorRT 推理，需配合 TensorRT8.6.1.6

其他版本的cuda和cudnn一律不支持，除非你自己编译

[YaoFANGUK]

@2922832737C 你是4090 显卡吗？如果是的话，那跟我的问题是一样的。cuda 11.7 太早起，无法支持 4090下运行 pytorch

pytorch删了cuda11.7的，下载11.8的

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

注意--index-url https://download.pytorch.org/whl/cu118一定要写

[floqve]

4090显卡换成LAMA算法后 配置cuda11.8 +cuDNN8.6。或者 cuda12.2 +cuDNN8.9 会报另外的错误 跑不起来

[floqve]

4090显卡换成LAMA算法后 配置cuda11.8 +cuDNN8.6。或者 cuda12.2 +cuDNN8.9 会报另外的错误 跑不起来

[u5207541]

cuda12.2 paddle 都不支持，当然跑不起来

[fliker09]

Found solution for this one. I had to use pip command instead of conda for installing PyTorch. After that ensure you are using the CuDNN as per instructions. In my first attempt I've used conda command, which installed PyTorch with CuDNN 8.7 bundled in. I matched it accordingly and it failed to work. Once I've re-created the env, used the pip command and used CuDNN 8.4.1 - it started working!

[daconghz]

@fliker09 Would you please write down here your detail install steps? Thanks.

[Ultraman6]

所以到底怎么办啊，40系显卡不配是吧？

[fliker09]

@daconghz just follow the instructions. Do NOT use the provided driver though! Get it from Nvidia (they provide PPA for Ubuntu). And as I mentioned above - install PyTorch using pip and NOT conda. @Ultraman6 no idea tbh, just try it out. Overall I am not happy with it - randomly failing to remove characters and also dropped frames, which completely messed up the sync between video and audio (I tried to recover - impossible, frames seems to have been dropped randomly).

[dmzlingyin]

有哪位大佬解决这个问题了吗? 我本地机器配置为 4090 显卡，cuda 12.2