paddleocr kie训练报错

[slightly0]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：Windows10
• 版本号/Version：x86_64
• Paddle：paddlepaddle-gpu2.3.2
• PaddleOCR： 2.6
• 问题相关组件/Related components：configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml
• 运行指令/Command Code：通过VScode运行，训练模型过程中报错
• 完整报错/Complete Error Message：
• Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Traceback (most recent call last): File "D:\Anaconda\envs\paddle\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "D:\Anaconda\envs\paddle\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "c:\Users\lenovo.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy\adapter/../..\debugpy\launcher/../..\debugpy__main.py", line 39, in cli.main() File "c:\Users\lenovo.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy\adapter/../..\debugpy\launcher/../..\debugpy/..\debugpy\server\cli.py", line 430, in main
  run() File "c:\Users\lenovo.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy\adapter/../..\debugpy\launcher/../..\debugpy/..\debugpy\server\cli.py", line 284, in run_file runpy.run_path(target, run_name="main") File "c:\Users\lenovo.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy_vendored\pydevd_pydevd_bundle\pydevd_runpy.py", line 321, in run_path return _run_module_code(code, init_globals, run_name, File "c:\Users\lenovo.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy_vendored\pydevd_pydevd_bundle\pydevd_runpy.py", line 135, in _run_module_code _run_code(code, mod_globals, init_globals, File "c:\Users\lenovo.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy_vendored\pydevd_pydevd_bundle\pydevd_runpy.py", line 124, in _run_code exec(code, run_globals) File "./main.py", line 13, in mymain() File ".\tools\train.py", line 202, in mymain main(config, device, logger, vdl_writer) File ".\tools\train.py", line 175, in main program.train(config, train_dataloader, valid_dataloader, device, model, File ".\tools\program.py", line 304, in train preds = model(batch) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "E:\jyjt\脚本测试文件\模型\PaddleOCR-release-2.6\ppocr\modeling\architectures\base_model.py", line 86, in forward x = self.backbone(x) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(inputs, kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "E:\jyjt\脚本测试文件\模型\PaddleOCR-release-2.6\ppocr\modeling\backbones\vqa_layoutlm.py", line 166, in forward x = self.model( File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, *kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(inputs, **kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddlenlp\transformers\layoutxlm\modeling.py", line 1040, in forward outputs = self.layoutxlm( File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call__ return self._dygraph_call_func(*inputs, *kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(inputs, **kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddlenlp\transformers\layoutxlm\modeling.py", line 925, in forward final_bbox = paddle.concat([bbox, visual_bbox], axis=1) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\tensor\manipulation.py", line 331, in concat return paddle.fluid.layers.concat(input=x, axis=axis, name=name) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\layers\tensor.py", line 343, in concat _C_ops.concat(input, out, 'axis', axis) OSError: (External) CUDA error(719), unspecified launch failure. [Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. Less common cases can be system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA, the process must be terminated and relaunched.] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:258) [operator < concat > error]

[andyjiang1116]

有更改配置文件吗？有的话，详细描述下你这边改了啥

[slightly0]

有更改配置文件吗？有话，详细描述下你改了啥

只修改了批次大小

[andyjiang1116]

训练命令是啥？paddle确保安装成功了吗

[slightly0]

训练命令是啥？paddle确保安装成功了吗

paddle安装成功的，使用vscode训练的，数据集也是按照配置文件里的路劲创建的

[andyjiang1116]

提供一下训练命令和配置文件吧

[slightly0]

提供一下训练命令和配置文件吧

[slightly0]

提供一下训练命令和配置文件吧

其他的都是默认设置

[andyjiang1116]

num_workers改成0

[slightly0]

num_workers改成0

改成0也会报错上面那个错QAQ

[andyjiang1116]

运行命令提供一下，windows只支持单卡训练

[slightly0]

我是通过VScode的debug运行的

[slightly0]

运行命令提供一下，windows只支持单卡训练

上面的图是我train.py的程序运行入口

[andyjiang1116]

按照文档在命令行执行训练指令

[slightly0]

按照文档在命令行执行训练指令 我只在文档上看到检测和部署命令 能给我发下相关文档吗

[andyjiang1116]

推理部署文档里面有给训练文档链接 https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/doc/doc_ch/kie.md

[slightly0]

推理部署文档里面有给训练文档链接 https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/doc/doc_ch/kie.md

[2022/09/13 14:23:01] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 19 iterations Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Error: ../paddle/phi/kernels/funcs/elementwise_functor.h:545 Assertion b != 0 failed. InvalidArgumentError: Integer division by zero encountered in (floor) divide. Please check the input value. Traceback (most recent call last): File "tools/train.py", line 209, in main(config, device, logger, vdl_writer) File "tools/train.py", line 175, in main program.train(config, train_dataloader, valid_dataloader, device, model, File "E:\吉元\脚本测试文件\模型\PaddleOCR-release-2.6\tools\program.py", line 304, in train preds = model(batch) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "E:\吉元\脚本测试文件\模型\PaddleOCR-release-2.6\ppocr\modeling\architectures\base_model.py", line 86, in forward x = self.backbone(x) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(inputs, kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, kwargs) File "E:\吉元\脚本测试文件\模型\PaddleOCR-release-2.6\ppocr\modeling\backbones\vqa_layoutlm.py", line 166, in forward x = self.model( File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, *kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(inputs, kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddlenlp\transformers\layoutxlm\modeling.py", line 1040, in forward outputs = self.layoutxlm( File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, *kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(inputs, **kwargs) File "D:\Anaconda\envs\paddle\lib\site-packages\paddlenlp\transformers\layoutxlm\modeling.py", line 911, in forward visual_bbox = paddle.stack( File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\tensor\manipulation.py", line 903, in stack return layers.stack(x, axis, name) File "D:\Anaconda\envs\paddle\lib\site-packages\paddle\fluid\layers\nn.py", line 10397, in stack return _C_ops.stack(x, 'axis', axis) OSError: (External) CUDA error(719), unspecified launch failure. [Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. Less common cases can be system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA, the process must be terminated and relaunched.] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:251) [operator < stack > error]

[slightly0]

推理部署文档里面有给训练文档链接 https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/doc/doc_ch/kie.md 报的还是之前的错误

[andyjiang1116]

先测试下paddle能否正常运行

import paddle
paddle.utils.run_check()

如果正常，提供一下完整的配置文件和运行命令

[slightly0]

先测试下paddle能否正常运行

可以正常运行

[slightly0]

运行命令

[littletomatodonkey]

你好，看你配置写的是layoutlm，但是代码报错走到了layoutxlm.py文件中了，这里建议安装下paddlenlp：https://paddleocr.bj.bcebos.com/ppstructure/whl/paddlenlp-2.3.0.dev0-py3-none-any.whl

同时拉下最新代码和数据看下：https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml

[slightly0]

你好，看你配置写的是layoutlm，但是代码报错了layoutxlm.py文件中了，这里建议安装下paddlenlp：https://paddleocr.bj.bcebos.com/ppstructure/whl/paddlenlp-2.3.0 。 dev0-py3-none-any.whl

拉下最新代码和数据看下：https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml

好的 我拉下最新的代码看下

[slightly0]

你好，看你配置写的是layoutlm，但是代码报错了layoutxlm.py文件中了，这里建议安装下paddlenlp：https://paddleocr.bj.bcebos.com/ppstructure/whl/paddlenlp-2.3.0 。 dev0-py3-none-any.whl

拉下最新代码和数据看下：https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/kie/layoutlm_series/ser_layoutlm_xfund_zh.yml

最新的代码 报错还是一样 错误发生在paddle.tensor数据计算时报错，最终定位报错位置是在以下这个文件中

[littletomatodonkey]

你好，我这边训练比较正常，你可以拉下代码再试下，建议先不要修改配置文件，按照教程先把流程走通

• paddlenlp：https://paddleocr.bj.bcebos.com/ppstructure/whl/paddlenlp-2.3.0.dev0-py3-none-any.whl
• xfund数据地址： https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar
• paddlepaddle：2.3.1
• paddleocr：dygraph

[husiluanxiang]

你好，我这边训练比较正常，你可以拉下代码再试下，建议先不要修改配置文件，按照教程先把流程走通

• paddlenlp：https://paddleocr.bj.bcebos.com/ppstructure/whl/paddlenlp-2.3.0.dev0-py3-none-any.whl
• xfund数据地址： https://paddleocr.bj.bcebos.com/ppstructure/dataset/XFUND.tar
• paddlepaddle：2.3.1
• paddleocr：dygraph

我也是同样的错误 [2022/09/27 13:54:04] ppocr INFO: train dataloader has 149 iters [2022/09/27 13:54:04] ppocr INFO: valid dataloader has 50 iters [2022/09/27 13:54:04] ppocr INFO: During the training process, after the 0th iteration, an evaluation is run every 19 iterations Traceback (most recent call last): File "D:\QB\pypro\PaddleOCR\tools\train.py", line 202, in main(config, device, logger, vdl_writer) File "D:\QB\pypro\PaddleOCR\tools\train.py", line 175, in main program.train(config, train_dataloader, valid_dataloader, device, model, File "D:\QB\pypro\PaddleOCR\tools\program.py", line 320, in train loss = loss_class(preds, batch) File "D:\QB\Anaconda3\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(*inputs, kwargs) File "D:\QB\Anaconda3\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "D:\QB\pypro\PaddleOCR\ppocr\losses\vqa_token_layoutlm_loss.py", line 41, in forward loss = self.loss_class(active_output, active_label) File "D:\QB\Anaconda3\lib\site-packages\paddle\fluid\dygraph\layers.py", line 930, in call return self._dygraph_call_func(inputs, kwargs) File "D:\QB\Anaconda3\lib\site-packages\paddle\fluid\dygraph\layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "D:\QB\Anaconda3\lib\site-packages\paddle\nn\layer\loss.py", line 397, in forward ret = paddle.nn.functional.cross_entropy( File "D:\QB\Anaconda3\lib\site-packages\paddle\nn\functional\loss.py", line 1718, in cross_entropy if label_min < 0: File "D:\QB\Anaconda3\lib\site-packages\paddle\fluid\dygraph\varbase_patch_methods.py", line 669, in bool return self.nonzero() File "D:\QB\Anaconda3\lib\site-packages\paddle\fluid\dygraph\varbase_patch_methods.py", line 666, in nonzero return bool(np.all(tensor.array() > 0)) OSError: (External) CUDA error(719), unspecified launch failure. [Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. Less common cases can be system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA, the process must be terminated and relaunched.] (at ..\paddle\phi\backends\gpu\cuda\cuda_info.cc:258) 这个错误是和显卡有关吗？你们的程序运行是不是基于V100的基础来运行的