PaddlePaddle / PaddleSeg

Easy-to-use image segmentation library with awesome pre-trained model zoo, supporting wide-range of practical tasks in Semantic Segmentation, Interactive Segmentation, Panoptic Segmentation, Image Matting, 3D Segmentation, etc.
https://arxiv.org/abs/2101.06175
Apache License 2.0
8.69k stars 1.68k forks source link

测试官方用例,需要等待二十多分钟才开始训练 #932

Closed AlwaysGemini closed 1 year ago

AlwaysGemini commented 3 years ago

1)PaddlePaddle版本:2.0.1 2)CPU:预测若用CPU,请提供CPU型号,MKL/OpenBlas/MKLDNN/等数学库使用情况 3)GPU:GTX 3070,CUDA 10.1、CUDNN 7.6 4)系统环境:Windows 10,Python 3.6

训练信息 1)单机,单卡 2)显存信息:8G 3)Operator信息 运行官方用例后,需要等待二十分钟左右才会开始训练

`(paddleseg) G:\project\PythonProject\PaddleSeg>python train.py --config configs/quick_start/bisenet_optic_disc_512x512_1k.yml 'C:\Program' 不是内部或外部命令,也不是可运行的程序 或批处理文件。 2021-03-30 14:34:35 [INFO] ------------Environment Information------------- platform: Windows-10-10.0.19041-SP0 Python: 3.6.13 (default, Feb 19 2021, 05:17:09) [MSC v.1916 64 bit (AMD64)] Paddle compiled with cuda: True NVCC: Not Available cudnn: 7.6 GPUs used: 1 CUDA_VISIBLE_DEVICES: None GPU: ['GPU 0: GeForce RTX'] GCC: gcc (x86_64-posix-seh-rev0, Built by MinGW-W64 project) 8.1.0 PaddlePaddle: 2.0.1 OpenCV: 4.5.1 2021-03-30 14:34:35 [INFO] ---------------Config Information--------------- batch_size: 4 iters: 100000 learning_rate: decay: end_lr: 0 power: 0.9 type: poly value: 0.01 loss: coef:

1 1 1 1 1 types: ignore_index: 255 type: CrossEntropyLoss model: pretrained: null type: BiSeNetV2 optimizer: momentum: 0.9 type: sgd weight_decay: 4.0e-05 train_dataset: dataset_root: data/optic_disc_seg mode: train transforms: target_size: 512 512 type: Resize type: RandomHorizontalFlip type: Normalize type: OpticDiscSeg val_dataset: dataset_root: data/optic_disc_seg mode: val transforms: target_size: 512 512 type: Resize type: Normalize type: OpticDiscSeg W0330 14:34:35.139756 6028 device_context.cc:362] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.1, Runtime API Version: 10.1 W0330 14:34:35.139756 6028 device_context.cc:372] device: 0, cuDNN Version: 7.6. 2021-03-30 14:54:49 [INFO] [TRAIN] epoch=1, iter=10/100000, loss=2.3108, lr=0.009999, batch_cost=0.1401, reader_cost=0.00990, ips=28.5542 samples/sec | ETA 03:53:27 2021-03-30 14:54:51 [INFO] [TRAIN] epoch=1, iter=20/100000, loss=0.9214, lr=0.009998, batch_cost=0.1223, reader_cost=0.00000, ips=32.7068 samples/sec | ETA 03:23:47 2021-03-30 14:54:52 [INFO] [TRAIN] epoch=1, iter=30/100000, loss=0.6966, lr=0.009997, batch_cost=0.1224, reader_cost=0.00000, ips=32.6903 samples/sec | ETA 03:23:52 2021-03-30 14:54:53 [INFO] [TRAIN] epoch=1, iter=40/100000, loss=0.6035, lr=0.009996, batch_cost=0.1222, reader_cost=0.00000, ips=32.7353 samples/sec | ETA 03:23:34 2021-03-30 14:54:54 [INFO] [TRAIN] epoch=1, iter=50/100000, loss=0.5113, lr=0.009996, batch_cost=0.1225, reader_cost=0.00000, ips=32.6541 samples/sec | ETA 03:24:03 2021-03-30 14:54:56 [INFO] [TRAIN] epoch=1, iter=60/100000, loss=0.5026, lr=0.009995, batch_cost=0.1232, reader_cost=0.00000, ips=32.4717 samples/sec | ETA 03:25:11`

wuyefeilin commented 3 years ago

@AlwaysGemini 首次运行的话会先下载数据集,导致启动时间偏慢

AlwaysGemini commented 3 years ago

@AlwaysGemini 首次运行的话会先下载数据集,导致启动时间偏慢

不是第一次,是每一次运行都会这样

wuyefeilin commented 3 years ago

@AlwaysGemini 多试几个模型,看看会不会有类似的结果

AlwaysGemini commented 3 years ago

@AlwaysGemini 多试几个模型,看看会不会有类似的结果

都是这样,而且可以确定是在train中的137行cfg.model占用二十分钟的,我之前打断点,这个断点过二十分钟后,下一行就可以顺畅运行了

wuyefeilin commented 3 years ago

@AlwaysGemini 多试几个模型,看看会不会有类似的结果

都是这样,而且可以确定是在train中的137行cfg.model占用二十分钟的,我之前打断点,这个断点过二十分钟后,下一行就可以顺畅运行了

应该是GTX显卡上构建网络的问题了,你试试直接调用Conv2D算子会不会有类似的情况

AlwaysGemini commented 3 years ago

@AlwaysGemini 多试几个模型,看看会不会有类似的结果

都是这样,而且可以确定是在train中的137行cfg.model占用二十分钟的,我之前打断点,这个断点过二十分钟后,下一行就可以顺畅运行了

应该是GTX显卡上构建网络的问题了,你试试直接调用Conv2D算子会不会有类似的情况

刚刚测试了一下了,如你所说,调用Conv2D,会有这样的情况

wuyefeilin commented 3 years ago

@AlwaysGemini 多试几个模型,看看会不会有类似的结果

都是这样,而且可以确定是在train中的137行cfg.model占用二十分钟的,我之前打断点,这个断点过二十分钟后,下一行就可以顺畅运行了

应该是GTX显卡上构建网络的问题了,你试试直接调用Conv2D算子会不会有类似的情况

刚刚测试了一下了,如你所说,调用Conv2D,会有这样的情况

多谢反馈,我们会把情况反馈给相关的同学

lmmir commented 3 years ago

3060显卡? 3060 要用cuda11 以上版本 用10.2 会出现你描述的情况 我之前遇到过

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.