本地多卡训练时，程序终止，服务器用户退出

PaddlePaddle / Paddle

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）

http://www.paddlepaddle.org/

Apache License 2.0

22.2k stars 5.58k forks source link

本地多卡训练时，程序终止，服务器用户退出 #29334

Closed shinianzhihou closed 2 years ago

shinianzhihou commented 3 years ago

版本、环境信息： Paddle version: 1.8.5 Paddle With CUDA: True OS: debian buster/sid Python version: 3.6.11 CUDA version: 10.0.130 cuDNN version: 7 Nvidia driver version: 450.80.02
训练信息 1）单机多卡 2）显存增长到一定时，程序异常终止，服务器当前用户直接退出，尝试调小batchsize也无效。运行命令是： export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python pdseg/train.py --use_gpu \ --cfg ../configs/deeplab_res2net_vd.yaml\ --use_vdl \ --vdl_log_dir ../logs/deeplab_res2net_vd_resume10/ \ --log_steps 20 \ --use_mpio \ --do_eval \ SOLVER.LR 0.05 \ BATCH_SIZE 120 \ SOLVER.LR_WARMUP True \ SOLVER.LR_WARMUP_STEPS 100 \ TRAIN.SNAPSHOT_EPOCH 1 \ TRAIN.SYNC_BATCH_NORM True \ TRAIN.PRETRAINED_MODEL_DIR ./saved_model/deeplab_res2net_vd/10

paddle-bot-old[bot] commented 3 years ago

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

shinianzhihou commented 3 years ago

可能是mpio的问题，我关了--use_mpio 就行了，但是速度降了很多

shinianzhihou commented 3 years ago

又出现这个问题了，请问可能是什么原因呀？为什么会导致服务器的用户下线呢

wawltor commented 3 years ago

@wuyefeilin 帮这个同学看看这个问题哈

wuyefeilin commented 3 years ago

@shinianzhihou 请把issue提到https://github.com/PaddlePaddle/PaddleSeg中，方便我们跟踪哈。

有可能是因为资源超载引起的问题. cuDNN version: 7, 具体是7.几给一下哈

paddle-bot-old[bot] commented 2 years ago

Since you haven\'t replied for more than a year, we have closed this issue/pr. If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up. 由于您超过一年未回复，我们将关闭这个issue/pr。若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。