ExternalError: CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED.

jo-dean commented 1 year ago

问题确认 Search before asking

[X] 我已经查询历史issue，没有发现相似的bug。I have searched the issues and found no similar bug report.

Bug组件 Bug Component

No response

Bug描述 Describe the Bug

在跑pphuman时出现了以下问题:

video fps: 25, frame_count: 126 Thread: 0; frame id: 0 Traceback (most recent call last): File "/home//AI/PaddleDetection/deploy/pipeline/pipeline.py", line 1343, in main() File "/home/AI/PaddleDetection/deploy/pipeline/pipeline.py", line 1310, in main pipeline.run_multithreads() File "/home//AI/PaddleDetection/deploy/pipeline/pipeline.py", line 180, in run_multithreads self.predictor.run(self.input) File "/home/AI/PaddleDetection/deploy/pipeline/pipeline.py", line 534, in run self.predict_video(input, thread_idx=thread_idx) File "/home/AI/PaddleDetection/deploy/pipeline/pipeline.py", line 754, in predict_video res = self.mot_predictor.predict_image( File "/home/AI/PaddleDetection/deploy/pptracking/python/mot_sde_infer.py", line 538, in predict_image result = self.predict() File "/home//AI/PaddleDetection/deploy/pptracking/python/det_infer.py", line 165, in predict self.predictor.run() OSError: In user code:

File "tools/export_model.py", line 111, in <module>
  main()
File "tools/export_model.py", line 107, in main
  run(FLAGS, cfg)
File "tools/export_model.py", line 73, in run
  trainer.export(FLAGS.output_dir)
File "/paddle/pipeline/PaddleDetection/ppdet/engine/trainer.py", line 775, in export
  save_dir)
File "/paddle/pipeline/PaddleDetection/ppdet/engine/trainer.py", line 752, in _get_infer_cfg_and_input_spec
  input_spec, static_model.forward.main_program,
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 563, in main_program
  concrete_program = self.concrete_program
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 479, in concrete_program
  return self.concrete_program_specify_input_spec(input_spec=None)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 517, in concrete_program_specify_input_spec
  *desired_input_spec)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 427, in get_concrete_program
  concrete_program, partial_program_layer = self._program_cache[cache_key]
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 723, in __getitem__
  self._caches[item] = self._build_once(item)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 714, in _build_once
  **cache_key.kwargs)
File "<decorator-gen-99>", line 2, in from_func_spec

File "/usr/local/lib/python3.7/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
  return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
  return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 662, in from_func_spec
  outputs = static_func(*inputs)
File "/tmp/tmp6bvkfcyf.py", line 100, in forward
  false_fn_5, (), (inputs, self), (out,))
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
  out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
  return true_fn(*true_args) if pred else false_fn(*false_args)
File "/tmp/tmp6bvkfcyf.py", line 83, in false_fn_5
  for_loop_body_0, [inputs_list, __for_loop_var_index_0])
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 45, in convert_while_loop
  loop_vars = _run_py_while(cond, body, loop_vars)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 59, in _run_py_while
  loop_vars = body(*loop_vars)
File "/tmp/tmp6bvkfcyf.py", line 78, in for_loop_body_0
  dy2static.convert_call(self.get_pred)())
File "/paddle/pipeline/PaddleDetection/ppdet/modeling/architectures/yolo.py", line 128, in get_pred
  return self._forward()
File "/tmp/tmpcqaxn1fy.py", line 114, in _forward
  __return_value_0, neck_feats, self), (__return_value_0,))
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
  out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
  return true_fn(*true_args) if pred else false_fn(*false_args)
File "/paddle/pipeline/PaddleDetection/ppdet/modeling/architectures/yolo.py", line 96, in _forward
  yolo_head_outs = self.yolo_head(neck_feats)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/tmp/tmp1s9lgv4m.py", line 28, in forward
  (__return_value_2, feats, self), (__return_value_2,))
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse
  out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse
  return true_fn(*true_args) if pred else false_fn(*false_args)
File "/paddle/pipeline/PaddleDetection/ppdet/modeling/heads/ppyoloe_head.py", line 219, in forward
  return self.forward_eval(feats)
File "/tmp/tmpg7ah4pu2.py", line 72, in forward_eval
  for_loop_body_4, [feats, i, __for_loop_var_index_6])
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 45, in convert_while_loop
  loop_vars = _run_py_while(cond, body, loop_vars)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 59, in _run_py_while
  loop_vars = body(*loop_vars)
File "/paddle/pipeline/PaddleDetection/ppdet/modeling/heads/ppyoloe_head.py", line 196, in forward_eval
  cls_logit = self.pred_cls[i](self.stem_cls[i](feat, avg_feat) +
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/paddle/pipeline/PaddleDetection/ppdet/modeling/heads/ppyoloe_head.py", line 43, in forward
  weight = F.sigmoid(self.fc(avg_feat))
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
  return self._dygraph_call_func(*inputs, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
  outputs = self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/nn/layer/conv.py", line 677, in forward
  use_cudnn=self._use_cudnn)
File "/usr/local/lib/python3.7/site-packages/paddle/nn/functional/conv.py", line 148, in _conv_nd
  type=op_type, inputs=inputs, outputs=outputs, attrs=attrs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
  return self.main_program.current_block().append_op(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 3184, in append_op
  attrs=kwargs.get("attrs", None))
File "/usr/local/lib/python3.7/site-packages/paddle/fluid/framework.py", line 2224, in __init__
  for frame in traceback.extract_stack():

ExternalError: CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED. 
  [Hint: 'CUDNN_STATUS_NOT_SUPPORTED'.  The functionality requested is not presently supported by cuDNN.  ] (at ../paddle/phi/kernels/fusion/gpu/conv_fusion_kernel.cu:616)
  [operator < conv2d_fusion > error]

复现环境 Environment

OS linux
paddlepaddle-gpu 2.5.1.post117
cuda:11.8
cudnn:8.8.1

Bug描述确认 Bug description confirmation

[X] 我确认已经提供了Bug复现步骤、代码改动说明、以及环境信息，确认问题是可以复现的。I confirm that the bug replication steps, code change instructions, and environment information have been provided, and the problem can be reproduced.