search

PaddlePaddle / PaddleDetection

Object Detection toolkit based on PaddlePaddle. It supports object detection, instance segmentation, multiple object tracking and real-time multi-person keypoint detection.
Apache License 2.0
12.75k stars 2.88k forks source link

training speed #8731

Open terryll opened 11 months ago

terryll commented 11 months ago

问题确认 Search before asking

请提出你的问题 Please ask your question

The training is not using all the availible resources in the system. The training commond is: python -u tools/train.py -c configs/ssd/ssdlite_mobilenet_v3_small_320_coco.yml --use_vdl=true --vdl_log_dir=vdl_dir/scalar --eval -o use_gpu=true Resource used are: GPU: 12-42% cpu:: 18-22% memory: 60% Disk: 6% The training eta for 100 epoch is 100. eta is 4 days. Is there way to set up the training to use more resource to train faster. Thanks in advance.

LokeZhou commented 7 months ago

Can use mixed precision training，set use_amp: O2 in ssdlite_mobilenet_v3_small_320_coco.yml or cmd --use_amp O2