PaddlePaddle / PaddleX

PaddlePaddle End-to-End Development Toolkit(飞桨低代码开发工具)
Apache License 2.0
4.6k stars 906 forks source link

训练异常终止 #1680

Open zealot9527 opened 1 year ago

zealot9527 commented 1 year ago

Checklist:

  1. 查找历史相关issue寻求解答
  2. 翻阅FAQ常见问题汇总和答疑
  3. 确认bug是否在新版本里还未修复
  4. 如果bug是由PaddleX API 2.0导致,且该bug在develop分支里已修复,参考FAQ Q4替换内置PaddleX API

描述问题

复现

  1. 请提供您出现的报错信息及相关log(log的查找见 FAQ Q2) File "D:\Program Files\PaddleX_GUI_2.1.0_win10\paddle\fluid\initializer.py", line 719, in call stop_gradient=True) File "D:\Program Files\PaddleX_GUI_2.1.0_win10\paddle\fluid\framework.py", line 3167, in append_op kwargs.get("stop_gradient", False)) File "D:\Program Files\PaddleX_GUI_2.1.0_win10\paddle\fluid\dygraph\tracer.py", line 45, in trace_op not stop_gradient) SystemError: (Fatal) Operator gaussian_random raises an class thrust::system::system_error exception. The exception content is :parallel_for failed: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device. (at ..\paddle\fluid\imperative\tracer.cc:221)

  2. 请提供您使用的GUI版本号 2.1.0

  3. 请提供您使用的操作系统信息,如Linux/Windows/MacOS windows11 显卡750Ti cuda11.5 cudnn8.3

  4. 请问您使用的CUDA/cuDNN的版本号是?

lailuboy commented 1 year ago

尝试降低一下Cuda版本,11.2

fenlier1 commented 1 year ago

Process Process-1:14: Traceback (most recent call last): File "multiprocessing\process.py", line 297, in _bootstrap File "multiprocessing\process.py", line 99, in run File "paddlexui\pms\model_tasks\tasks.py", line 58, in _call_paddlex_train File "D:\paddleX\PaddleX_GUI_2.1.0_win10\paddlex__init.py", line 20, in from . import cv File "D:\paddleX\PaddleX_GUI_2.1.0_win10\paddlex\cv__init__.py", line 15, in from . import models File "D:\paddleX\PaddleX_GUI_2.1.0_win10\paddlex\cv\models\init__.py", line 17, in from .detector import * File "D:\paddleX\PaddleX_GUI_2.1.0win10\paddlex\cv\models\detector.py", line 316 'ESNet' in train self.backbone_name)) ^ SyntaxError: invalid syntax

fenlier1 commented 1 year ago

1.请提供您出现的报错信息及相关log Process Process-1:14: Traceback (most recent call last): File "multiprocessing\process.py", line 297, in bootstrap File "multiprocessing\process.py", line 99, in run File "paddlexui\pms\model_tasks\tasks.py", line 58, in call_paddlex_train File "D:\paddleX\PaddleX_GUI_2.1.0_win10\paddlex_init.py", line 20, in from . import cv File "D:\paddleX\PaddleX_GUI_2.1.0_win10\paddlex\cv_init.py", line 15, in from . import models File "D:\paddleX\PaddleX_GUI_2.1.0_win10\paddlex\cv\modelsinit.py", line 17, in from .detector import * File "D:\paddleX\PaddleX_GUI_2.1.0win10\paddlex\cv\models\detector.py", line 316 'ESNet' in train self.backbone_name)) ^ SyntaxError: invalid syntax 2.请提供您使用的GUI版本号 2.1.0 3.请提供您使用的操作系统信息 Windows10 显卡1660ti 4.请问您使用的CUDA/cuDNN的版本号是? CUDA11.2 CUDNN8.1.0